{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Advanced usage tutorial\n", "\n", "This notebook assumes that you are already familiar with the hiresprv Python package and have run through the process described in the basic data reduction tutorial notebook. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook is meant to be a template to set up your own processing using the code snippets provided here. The notebook may not run exactly as presented if you attempt to run the full notebook directly. The code has not been rigorously tested from within a notebook environemnt.\n", "\n", "\n", "\n", "The notebook is kept with the HIRES PRV Python access toolkit in GitHub: https://github.com/Caltech-IPAC/hiresprv\n", "\n", "You can download and interact with your own copy of this notebook [here](../_static/Advanced_Usage.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Login\n", "\n", "As described in the basic data reduction tutorial we always need to login to our KOA account first." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KOA userid: koaadmin\n", "KOA Password: ········\n", "Successful login as koaadmin\n" ] } ], "source": [ "from IPython.display import IFrame\n", "from hiresprv.auth import login\n", "\n", "login('prv.cookies')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ancillary Tools\n", "\n", "### Workspace Directory Listing\n", "\n", "We can get a list of all the downloadable files in the workspace. There are utility functions (below) for quickly downloading the most-useful final products but we might want to see the intermediate products to evaluate/diagnose the processing. See http://caltech-ipac.github.io/hiresprv/outputs.html#workspace-directory-structure for a description of the files availble in each directory." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"database\": \"prvState.db\",\n", " \"deblazed\": [\n", " \"deblazed/r20091231.160.fits\",\n", " \"deblazed/r20130629.110.fits\",\n", " \"deblazed/r20091231.287.fits\",\n", " \"deblazed/r20130912.73.fits\",\n", " \"deblazed/r20130912.1.fits\",\n", " \"deblazed/r20130912.128.fits\",\n", " \"deblazed/r20150606.157.fits\",\n", " \"deblazed/r20091231.225.fits\",\n", " \"deblazed/r20130912.163.fits\",\n", " \"deblazed/r20091231.189.fits\",\n", " \"deblazed/r20130629.92.fits\",\n", " \"deblazed/r20150606.97.fits\",\n", " \"deblazed/r20130912.152.fits\",\n", " \"deblazed/r20091231.214.fits\",\n", " \"deblazed/r20150606.166.fits\",\n", " \"deblazed/r20091231.67.fits\",\n", " \"deblazed/r20130912.119.fits\",\n", " \"deblazed/r20091231.151.fits\",\n", " \"deblazed/r20130912.130.fits\",\n", " \"deblazed/r20150606.104.fits\",\n", " \"deblazed/r20091231.276.fits\",\n", " \"deblazed/r20091231.191.fits\",\n", " \"deblazed/r20130912.82.fits\",\n", " \"deblazed/r20130629.19.fits\",\n", " \"deblazed/r20091231.178.fits\",\n", " \"deblazed/r20091231.133.fits\",\n", " \"deblazed/r20130629.108.fits\",\n", " \"deblazed/r20130912.11.fits\",\n", " \"deblazed/r20130629.3.fits\",\n", " \"deblazed/r20091231.96.fits\",\n", " \"deblazed/r20091231.102.fits\",\n", " \"deblazed/r20150606.66.fits\",\n", " \"deblazed/r20091231.3.fits\",\n", " \"deblazed/r20091231.149.fits\",\n", " \"deblazed/r20091231.247.fits\",\n", " \"deblazed/r20150606.135.fits\",\n", " \"deblazed/r20130912.101.fits\",\n", " \"deblazed/r20130629.96.fits\",\n", " \"deblazed/r20150606.129.fits\",\n", " \"deblazed/r20130912.156.fits\",\n", " \"deblazed/r20150606.162.fits\",\n", " \"deblazed/r20091231.210.fits\",\n", " \"deblazed/r20150606.93.fits\",\n", " \"deblazed/r20091231.155.fits\",\n", " \"deblazed/r20130629.114.fits\",\n", " \"deblazed/r20091231.283.fits\",\n", " \"deblazed/r20091231.164.fits\",\n", " \"deblazed/r20130912.5.fits\",\n", " \"deblazed/r20130912.77.fits\",\n", " \"deblazed/r20091231.221.fits\",\n", " \"deblazed/r\n", "... output truncated ...\n" ] } ], "source": [ "import json\n", "import os\n", "from hiresprv.download import Download\n", "\n", "data = Download('prv.cookies', './', debug=True)\n", "\n", "listing = data.directory_listing()\n", "prtstring = json.dumps(listing, indent=4, sort_keys=True)[:2000]\n", "prtstring += \"\\n... output truncated ...\"\n", "\n", "print(prtstring)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### File Download\n", "\n", "Any of the files in the listing can be retrieved by name. We specify the output directory when initializing the Download object." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'status': 'ok', 'msg': ''}\n" ] } ], "source": [ "rtn = data.download(\"deblazed/r20091231.247.fits\")\n", "\n", "print(rtn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Augmented Processing\n", "\n", "There are no tunable parameters associated with the HIRES PRV pipeline. However, you can affect the processing in a couple of places by \"removing\" files (usually because there is something suspect about them). For instance, a bad B-star observation can adversely affect the stellar template calculation so you may want to rerun the template building without it. We don't actually delete the file; we deactivate it but leave it in place in case you change your mind.\n", "\n", "Similarly, in making the final RV curve you may decide to remove one or more of the reduced RV measurements.\n", "\n", "\n", "### Deactivating Files\n", "\n", "Sometimes it turns out that a data file is suboptimal and should probably be removed from the processing. It might be a B-star observation bracketing a template measurement or one of the RV observations that should be removed from an RV curve.\n", "\n", "The PRV service takes responsibility for remembering what files have been processed and for providing a mechanism (the \"DEACTIVATED\" column in the database) for \"turning off\" specific files. It does not automatically redo all the affected downstream processing. That is left up to the user so it is best to take care of all of that as soon as possible.\n", "\n", "In this example, we will turn off one of the B-star observations. This requires regenerating the template(s) associated with that observation:\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "status= ok\n", "msg= Script running in background. Consult monitor for status.\n", "None\n" ] } ], "source": [ "from hiresprv.idldriver import Idldriver\n", "\n", "idl = Idldriver('prv.cookies')\n", "\n", "rtn = idl.run_script(\"\"\"\n", "deactivate r20091231.77\n", "template 185144 20091231\n", "\"\"\")\n", "\n", "print(rtn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having regenerated the template, we must therefore regenerate the downstream data (RVs and RV curve). The underlying processing code will not regenerate the reduced RV data if that file already exists so we need to explicitly deactivate the reduced RV files if we intend to re-calculate them:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "status= ok\n", "msg= Script running in background. Consult monitor for status.\n", "None\n" ] } ], "source": [ "from hiresprv.idldriver import Idldriver\n", "\n", "idl = Idldriver('prv.cookies')\n", "\n", "rtn = idl.run_script(\"\"\"\n", "deactivate vdaa185144_r20091231.72\n", "deactivate vdaa185144_r20091231.73\n", "deactivate vdaa185144_r20091231.74\n", "deactivate vdaa185144_r20150606.145\n", "deactivate vdaa185144_r20150606.146\n", "deactivate vdaa185144_r20150606.147\n", "rv 185144 r20091231.72\n", "rv 185144 r20091231.73\n", "rv 185144 r20091231.74\n", "rv 185144 r20150606.145\n", "rv 185144 r20150606.146\n", "rv 185144 r20150606.147\n", "rvcurve 185144\n", "\"\"\")\n", "\n", "print(rtn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After further consideration, we decide that the second RV measurement was of inferior quality, so we deactivate it, too. You can find the name of the reduced RV file by listing out the full database again or by doing a database search (if you look a the file name, it follows a simple pattern):" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DATEOBTYPEFILENAMETARGETBJDBCVEL
00Reduced RVvdaa185144_r20091231.72rv data2440000.00.0
10Reduced RVvdaa185144_r20091231.73rv data2440000.00.0
20Reduced RVvdaa185144_r20091231.74rv data2440000.00.0
30Reduced RVvdaa185144_r20150606.145rv data2440000.00.0
40Reduced RVvdaa185144_r20150606.146rv data2440000.00.0
50Reduced RVvdaa185144_r20150606.147rv data2440000.00.0
\n", "
" ], "text/plain": [ " DATE OBTYPE FILENAME TARGET BJD BCVEL\n", "0 0 Reduced RV vdaa185144_r20091231.72 rv data 2440000.0 0.0\n", "1 0 Reduced RV vdaa185144_r20091231.73 rv data 2440000.0 0.0\n", "2 0 Reduced RV vdaa185144_r20091231.74 rv data 2440000.0 0.0\n", "3 0 Reduced RV vdaa185144_r20150606.145 rv data 2440000.0 0.0\n", "4 0 Reduced RV vdaa185144_r20150606.146 rv data 2440000.0 0.0\n", "5 0 Reduced RV vdaa185144_r20150606.147 rv data 2440000.0 0.0" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from hiresprv.database import Database\n", "import pandas as pd\n", "\n", "state = Database('prv.cookies')\n", "\n", "url = state.search(sql=\"select DATE, OBTYPE, FILENAME, TARGET, BJD, BCVEL from FILES where TARGET like 'rv data';\")\n", "\n", "df = pd.read_html(url, header=0)[0]\n", "df.head(15)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "status= ok\n", "msg= Script running in background. Consult monitor for status.\n", "None\n" ] } ], "source": [ "from hiresprv.idldriver import Idldriver\n", "\n", "idl = Idldriver('prv.cookies')\n", "\n", "rtn = idl.run_script(\"\"\"\n", "deactivate vdaa185144_r20091231.73\n", "rvcurve 185144\n", "\"\"\")\n", "\n", "print(rtn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So now we have only five reduced RV measurements and they were reduced with a slightly different template measurement. This is reflected in the RV curve, which you can compare to the original:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BJD_TDB,RV,RV_ERR,BC,ADU,CHI2\n", "15196.69208800001,-2.176984931266687,0.782362,-4620.095214843750,52362,1.04989\n", "15196.69329199987,2.082068283340939,0.778802,-4620.320800781250,48950,1.05804\n", "17180.10972899990,-2.351030457353758,0.769511,3189.794921875000,55029,1.10542\n", "17180.11030799989,1.241724284090688,0.678055,3189.327880859375,55593,1.10771\n", "17180.11088699987,0.9359797393606732,0.686433,3188.863037109375,48769,1.10120\n" ] } ], "source": [ "from hiresprv.download import Download\n", "\n", "data = Download('prv.cookies', './')\n", "\n", "rtn = data.rvcurve('185144')\n", "\n", "with open('vst185144.csv', 'r') as file:\n", " for line in file:\n", " print(line, end='')\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }