Python API¶

These modules are arranged here in roughly the order they are normally used. For instance, the first step always has to be a KOA login.

The HIRES PRV pipeline is used to retrieve user-specified radial velocity data from the KOA archive to a user-specific pipeline server workspace, automatically perform raw data reduction (converting the 2D spectral files to 1D spectra) and then run user-defined reduction scripts to ultimately generate RV curves for specific sky objects.

The data is retrieved a day at a time (a full night’s data is needed for the raw reduction). Metadata for all the files retrieved is stored in a workspace/user-specific database table, which is user-searchable and provides the information needed to construct reduction scripts.

Both the archive retrieval and reduction steps can be quite lengthy (often hours), so this is done in background and there are tools for monitoring the state of the processing.

Authentication¶

The hiresprv.login module initializes an account on the PRV pipeline server for reduction of radial velocity data.

hiresprv.auth.login(cookiepath)[source]¶

The login function prompts for an authorized KOA user ID and password, then sets up a workspace (or connects to an existing workspace) on the PRV pipeline server for that user.

Parameters: cookiepath (string) – a file path provided by the user to save returned cookie which is needed for the subsequent PRV operations.

Example

>>> import hiresprv.auth
>>> hiresprv.auth.login(cookiepath)
# program will prompt for userid and password

Data Transfer and Reduction¶

The hiresprv.archive module handles data transfers from the Keck Observatory Archive (KOA) archive into the user workspace on the server and the reduction from 2D CCD images to 1D spectra.

class hiresprv.archive.Archive(cookiepath, **kwargs)[source]¶

The Archive class provides KOA archive access functions for the HIRES PRV pipeline reduction service.

The user’s KOA credentials (given at login) are used to search KOA for nights containing HIRES-PRV compatible data. Matching data are copied to the user’s workspace and raw reduction (conversion to 1D spectra, barycentric correction, file organization, etc.) is performed. Results are logged in the workspace database table. All this is done in background; the search functions return almost immediately with an acknowledgement.

Because of pipeline requirements, data is always processed a full night at a time.

Parameters: cookiepath – full path to a cookie file saved by hiresprv.auth.login()

by_dates(dates)[source]¶

Constructs and submits a URL to the server for processing

This method receives an acknowledgement upon successful submission which means it has successfully authenticated the KOA user and can start the data search, download, and reduction.

Parameters

dates (string) – a date string or multiple date strings separated by comma or newline. Each date should be in to ‘yyyy-mm-dd’ format.

Returns

Status (‘ok’ or ‘error’) and a message string

”{status’:’ok’, ‘msg’:’Processing dates in background.’}” if successful

”{‘status’:’error’, ‘msg’:’Failed to connect to KOA’}” if submission failed

Return type

JSON structure

Example

>>> import hiresprv.archive
>>> srch = hiresprv.archive.Archive(cookiepath)
>>> multi_date_string = "2013-09-23,2013-09-25,2013-10-01"
>>> srch.by_dates(multi_date_string)

by_datefile(datefile)[source]¶

This method operates the same as by_dates method except it reads the dates string from a file containing a list of dates.

Parameters: datefile (string) – Path to a file containing more than one date. Each date sould be in the ‘yyyy-mm-dd’ format and separated by new line.
Returns: Same status and msg as hiresprv.archive.Archive.by_dates() method
Return type: JSON structure

Progress Monitor¶

Monitor the progress of active processing jobs.

class hiresprv.status.Status(cookiepath, **kwargs)[source]¶

The HIRES PRV processing involves transferring and reducing a large amount of data and can be quite lengthy. Therefore, most of this processing is done in background.

The hiresprv.status.Status class is used to check the state of the processing, to watch progress, or simply to check whether the workspace is busy. New processing will be rejected until the workspace is ready.

The PRV Status class initialization checks for the existence of a login cookie and connects the user to their workspace. Methods of this class return information on the current state.

Parameters: cookiepath (string) – full path to cookie file saved from hiresprv.auth.login()

generate_link()[source]¶

This method returns an HTML string that contains a link to start the real-time monitor in a separate window/tab. It is generally used in applications like Jupyter notebook where you don’t want the monitor embedded in the page.

Returns: HTML fragment to be embedding in page to provide access to real-time monitor page.
Return type: string

processing_status()[source]¶

This method returns a URL to a page displaying the progress of the current processing step. For archive retrieval this includes each file transfer and each raw reduction operation. For data reduction scripts, this includes the various steps in the IDL processing.

An attempt has been made to update the processing status every few seconds to a minute but a few operations will run longer.

Returns: URL to a web page summarizing the progress of the current processing steps
Return type: string

is_busy()[source]¶

Check if the workspace is currently busy processing.

Returns: JSON structure

cancel()[source]¶

Cancel the active job.

Returns: JSON structure

Observation Database¶

Access and query a user’s observation database.

class hiresprv.database.Database(cookiepath, **kwargs)[source]¶

Each workspace in HIRES PRV pipeline server contains a database listing all the data retrieved from KOA for the user.

The hiresprv.status.Database class provides methods for querying that database. This information is primarily used to plan reduction and analysis processing.

The initialization of database class loads the cookie file saved from HIRES PRV pipelone login, parse the cookie to look up the users workspace.

Parameters: cookiepath (string) – full path to a cookie file saved by hiresprv.auth.login()

search(**kwargs)[source]¶

The search method is the most general mechanism for querying a workspace database. The user has the freedom to provide a general SQL SELECT statement (specifying both the database columns to be retrieved and constraints on the records returned).

When the sql is blank (i.e.: ‘’), the entire database will be returned.

The user can also select an output format (html, csv or IPAC ASCII).

The output will be saved to a disk file and/or displayed in a browser depending on the format:

‘cvs’ format: save to disk, ‘html’ format: save to disk and display in browser, ‘IPAC’ ASCII format: save to disk and display in browser.

Parameters

sql (string) – (optional) fully qualified sql statement if empty string or parameter not set, the whole database table will be returned.
format (string) – (optional) string specifying the output format (‘html’|’csv’|’ipac’); the default is html
filepath (string) – (optional) full path where the file will be saved; if not provided, a URL string to an HTML view of the table.

Returns

URL to HTML table if filepath is not specified

Return type

string

target_list(**kwargs)[source]¶

Returns a list of all unique targets currently in the database.

Parameters: None –
Returns: URL to HTML table if filepath is not specified
Return type: string

target_info(target, **kwargs)[source]¶

This method retrieves the database records for all data pertaining to a specific target.

Parameters

target (string) – target name
format (string) – (optional) string specifying the output format (‘html’|’csv’|’ipac’); the default is html
filepath (string) – (optional) full path where the file will be saved; if not provided, a URL string to an HTML view of the table.

Returns

URL to HTML table if filepath is not specified

Return type

string

sqlite(filepath)[source]¶

Downloads the sqlite database file

Parameters: filepath (string) – path and filename to save the database file on the disk.

rv_observations(target)[source]¶

Return metadata for all RV observations associated with a given target

Parameters: target (string) – target name

static to_pandas(url)[source]¶

Convert URL for HTML table into a Pandas DataFrame

Parameters: url (string) – database search url (produced by Database methods like search or target_info)
Returns: DataFrame

IDL Driver¶

Drive the underlying IDL code

class hiresprv.idldriver.Idldriver(cookiepath, **kwargs)[source]¶

The principle processing of the HIRES PRV pipeline is done by a set of IDL scripts developed over several decades. This processing is quite intensive, takes a long time, and is run in the background.

The hiresprv.idldriver.Idldriver class provides functionality that allows the user to submit reduction scripts that are parsed and sent to the appropriate IDL functions on the server.

The idldriver class intialization checks for cookie indicating a previous login that connects to the user to a PRV pipeline workspace. This workspace is populated with data from the KOA Archive using the hiresprv.archive.Archive class methods.

Parameters: cookiepath – a full path to cookie file saved from hiresprv.auth.login()

run_script(script)[source]¶

This method is given a script of steps to run on the data in the user’s workspace. These steps include creating a template spectrum for a sky target, reducing specific radial velocity measurement(s) using such a template, and creating an RV curve from a set of reduced RV measurements.

Parameters: script (string) – script containing processing steps separated by newlines

Example

>>> from hiresprv.idldriver import Idldriver
>>> idl = Idldriver('prv.cookies')
>>> rtn = idl.run_script("""
template 185144 20091231
rv 185144 r20091231.72
rv 185144 r20091231.73
rv 185144 r20091231.74
rv 185144 r20150606.145
rv 185144 r20150606.146
rv 185144 r20150606.147
rvcurve 185144""")

Note

List of available commands for run_script:

‘template <object> <date>’ (create template)

‘rv <obsid> (calculate RV for single observation)’

‘rvcurve <object> (construct RV timeseries for an object)’

‘activate <filename> (activate or include file in analysis)’

‘deactivate <filename> (deactivate or exclude file from analysis)’

run_scriptfile(scriptfile)[source]¶

Same as hiresprv.idldriver.Idldriver.run_script() except takes a path to a file containing the script lines.

Parameters: scriptfile (string) – path to plain text file that will be read as a continuous string and used as input to the hiresprv.idldriver.Idldriver.run_script() method.

static create_rvscript(target, db)[source]¶

Create script to prcess all RV files associated with a given target

Parameters

target (string) – target string
db (hiresprv.database.Database) – Database object from your workspace

Returns

string

Retrieve Results¶

Download data from the remote workspace

class hiresprv.download.Download(cookiepath, localdir, **kwargs)[source]¶

The Download class provides methods for users to download individual files from their workspace in HIRES PRV pipeline server.

It validates user information (via cookie file), then contacts PRV Server to retrieve the requested file.

The initialization of database class loads the cookie file saved from HIRES PRV pipeline login, parse the cookie to look up the users workspace.

Parameters

cookiepath (string) – a full cookie file path saved from auth.Login.
localdir (string) – local output directory

directory_listing()[source]¶

This method returns a listing of all the downloadable files in the workspace.

Returns: Dictionary with one entry for each workspace data directory. The value of each item is an array of full file paths within each directory.
Return type: JSON structure

download(filename)[source]¶

This method downloads any file from the user’s workspace.

Parameters: filename (string) – workspace file name
Returns: structure indicating the status of the submission
Return type: JSON structure

rvcurve(objname)[source]¶

This method downloads a rvcurve csv file from the user’s workspace.

Parameters: objname (string) – object name; must match an entry in the ‘TARGET’ column of the workspace database
Returns: structure indicating the status of the submission
Return type: JSON structure

spectrum(fileid)[source]¶

This method downloads one or more extracted spectrum FITS files from the user’s workspace. The downloaded file will be a single FITS file if fileids is a single observation code.

Parameters: fileid (string) – one or more spectrum file names separated by comma or new line; must match entries in the ‘FILENAME’ column in the workspace database
Returns: structure indicating the status of the submission
Return type: JSON structure

Return Home