|
ENLIGHTEN
Cross-platform desktop GUI for Wasatch Photonics spectrometers
|
Encapsulates a single saved measurement from one spectrometer, comprising a ProcessedReading (optionally containing the original Reading object that generated it), metadata (Settings), as well as a ThumbnailWidget for display on the capture bar. More...
Public Member Functions | |
| __init__ (self, ctl=None, processed_reading=None, settings=None, source_pathname=None, timestamp=None, spec=None, measurement=None, d=None) | |
| There are three valid instantiation patterns: | |
| add_pathname (self, pathname) | |
| build_row (self, field) | |
| Generate a single row of output for row-ordered CSV files. | |
| clear (self) | |
| clone (self) | |
| Called by PluginWidget. | |
| csv_formatted (self, roi, prec, array, pixel, obey_roi=False) | |
| Used by save_csv_file_by_column and save_csv_file_by_row. | |
| delete (self, from_disk=False, update_parent=False) | |
| Release any resources associated with this Measurement. | |
| display (self) | |
| Display on the graph, if not already shown. | |
| dump (self) | |
| expand_template (self, template) | |
| Some GUI text fields allow the user to enter strings containining "macro
templates" which are dynamically expanded and evaluated at runtime. | |
| generate_basename (self) | |
| generate_id (self) | |
| generate_label (self) | |
| generate_today_dir (self) | |
| dict | get_all_metadata (self) |
| get_csv_data (self, pr) | |
| Used by save_csv_file_by_column and save_csv_file_by_row. | |
| get_extra_header_fields (self) | |
| get_metadata (self, field) | |
| This function is provided because legacy Dash and ENLIGHTEN saved row- ordered CSV files with very specific column headers and sequence which we don't want to casually break. | |
| has_component (self, component) | |
| Not currently used. | |
| id_callback (self, declared_match) | |
| Called (by way of ThumbnailWidget -> KnowItAll.Feature -> Measurements) when KnowItAll has generated a KnowItAll.DeclaredMatch for this Measurement. | |
| init_from_dict (self, d) | |
| This is experimental. | |
| interpolate (self, new_settings) | |
| Passed a SpectrometerSettings object (containing wavelengths, wavenumbers etc), return a copy of this Measurement's ProcessedReading which has been interpolated to the passed x-axis. | |
| is_displayed (self) | |
| rename_files (self) | |
| The measurement has been relabled (say, "cyclohexane"). | |
| replace_processed_reading (self, pr) | |
| We presumably loaded a measurement from disk, reprocessed it, and are now replacing the contents of the Measurement object with the reprocessed spectra, preparatory to re-saving (with a new timestamp and measurement_id). | |
| save (self, resave=False) | |
| save_csv_file (self, resave=False) | |
| save_csv_file_by_column (self, use_basename=False, ext="csv", delim=",", include_header=True, include_metadata=True, resave=False) | |
| Save the Measurement in a CSV file with the x-axis in one column, spectra in the next column and so on (similar layout as the Excel output). | |
| save_csv_file_by_row (self, resave=False) | |
| Save a spectrum in CSV format with the whole spectrum on one line, such that multiple acquisitions could be appended with one line per spectrum. | |
| save_dx_file (self, use_basename=False, resave=False) | |
| save_excel_file (self, resave=False) | |
| Save the spectra in xls format (currently, one worksheet per x-axis) | |
| save_json_file (self, use_basename=False, resave=False) | |
| Save the Measurement in a JSON file for simplified programmatic parsing. | |
| save_spc_file (self, use_basename=False, resave=False) | |
| save_txt_file (self, use_basename=False, resave=False) | |
| This is essentially the same as column-ordered CSV, but with no metadata, no header row and no commas. | |
| to_dict (self) | |
| Express the current Measurement as a single JSON-compatible dict. | |
| to_json (self) | |
| Render the current Measurement to JSON. | |
| update_label (self, label, manual=False) | |
| verify_pathname (self, pathname, resave=False) | |
| write_processed_reading_lines (self, csv_writer) | |
| For row-ordered CSVs, output any ProcessedReading fields which have been selected. | |
| write_row (self, csv_writer, field) | |
| write_x_axis_lines (self, csv_writer) | |
| For row-ordered CSVs, output any x-axis fields that have been selected. | |
Static Public Member Functions | |
| generate_dash_file_header (serial_numbers) | |
| generate a legacy Dash file header with one or more serial numbers | |
Public Attributes | |
| bool | appending = False |
| baseline_correction_algo = None | |
| str | basename = None |
| ctl = ctl | |
| declared_match = None | |
| int | declared_score = 0 |
| label = None | |
| str | measurement_id = None |
| dict | metadata = {} |
| str | note = "" |
| dict | pathname_by_ext = {} |
| str | plugin_name = "" |
| str | prefix = "" |
| processed_reading = None | |
| bool | renamed_manually = False |
| bool | roi_active = False |
| settings = None | |
| source_pathname = None | |
| spec = None | |
| str | suffix = "" |
| technique = None | |
| thumbnail_widget = None | |
| timestamp = None | |
Static Public Attributes | |
| list | CSV_HEADER_FIELDS |
| These appear in legacy saved spectra files as-written (Dash format), so don't casually screw with them (could break customer applications). | |
| CSV_HEADER_FIELDS_SET = set(CSV_HEADER_FIELDS) | |
| list | EXTRA_HEADER_FIELDS |
| These fields weren't in the original Dash file format, so only use for the new "column-ordered" formats (including 'export') | |
| EXTRA_HEADER_FIELDS_SET = set(EXTRA_HEADER_FIELDS) | |
| list | ROW_ONLY_FIELDS = ['Blank', 'Line Number'] |
| These CSV_HEADER_FIELDS only need to be used for row-ordered files. | |
Encapsulates a single saved measurement from one spectrometer, comprising a ProcessedReading (optionally containing the original Reading object that generated it), metadata (Settings), as well as a ThumbnailWidget for display on the capture bar.
Note that other than the ProcessedReading, SpectrometerApplicationState is NOT retained in the Measurement.
A Measurement object is created when:
If a Measurement is loaded from disk, it will not contain the original Reading object. Also, the metadata may be limited.
If a Measurement is generated live, it will also have a reference to the spectrometer which generated it.
Regardless of whether the Measurement was generated from a live capture (Acquire) or loaded from disk, a ThumbnailWidget will be generated (not necessarily instantly) for visualization.
As some Measurements may be displayed in different x-axis coordinates, we need to store the "current / selected / displayed" x-axis for the Measurement. I'm tentatively thinking to do that in the ThumbnailWidget, as it's kind of an attribute of the trace, but we'll see.
I am not sure whether we really need separate Thumbnail and ThumbnailWidget classes, other than that they are literally very different things (the one is a pyqtgraph export, the other is a QWidget), even though one is typically displayed within the other.
Given that there are multiple ways to create a Measurement, with somewhat different attributes and controlled background timing, a MeasurementFactory is provided to separately encapsulate the process of construction.
Each Measurement has a .measurement_id, which is permanent. The id is used as the default label attribute (both for file basenames and for on-screen display), but the label may be subsequently changed by the user.
There is no simple .pathname attribute for the Measurement, as one Measurement may have been saved to any or all of several different file types.
AT PRESENT, it is assumed that the creation of a Measurement will include saving the Measurement to one or more output files at creation, but this is not a long-term requirement (we may decide to support "session traces" which are never actually persisted to disk; Spectrasuite and OceanView have these).
There is a .pathnames set which aggregates all the pathnames which are known related to this Measurement, though it is not guaranteed complete in the case of loaded files from other sessions. (Ideally all such files would contain the original measurement_id somewhere within them, though not necessarily in the pathname.)
If a Measurement has been deserialized from disk, it will have a .source_pathname attribute to indicate the source file from which it was instantiated.
Note that the application caps how many Measurements are visible in the Thumbnail bar at any given time, and currently ENLIGHTEN's file-management operations (rename, delete etc) only function on visible Thumbnails, so if you're streaming vast BatchCollections to disk such that they get rotated out of our buffer, you'll have to rename / delete them through other means.
Renaming Measurements is a potentially thorny issue. Early versions of ENLIGHTEN renamed the output file(s) when you changed the label of on-screen thumbnail, and customers wish to retain that ability (ticket from 2019-07-19).
However, there is all kinds of ugliness down this hole:
My current decision is that we will retain the historical ability to rename any file(s) saved from the given Measurement IFF the files were CREATED by the CURRENT ENLIGHTEN session; OR if the spectrum was loaded from a single file (not an export or appended collection).
This distinction is made when Measurements are loaded or saved, and tracked via a renameable_files list. It's not as simple as checking whether the spectrum was created or loaded, because it matters what type of file it was loaded from, or which type of file it was saved to.
A consequence of this design is that if you initially have CSV and XLS files saved, you can rename the thumbnail to "apple" and both CSV and XLS files will be renamed to apple.csv and apple.xls. However, if you then clear your thumbnails (or quit / relaunch ENLIGHTEN) and load apple.csv from disk, if you then rename the file to banana.csv, it will NOT likewise rename apple.xls to banana.xls.
Also, relabeling the Measurement on-screen, and even renaming the underlying file artifacts DOES NOT re-write or update the file contents. The "label" field in the file metadata is not updated. It would not be particularly hard to re-save the file, but this seems like it would have many risks if a newer version of ENLIGHTEN changed the file format, or if there had been extra data in the file (ignored during a load operation, or added by the user post-save) which would then be destroyed.
Originally, ENLIGHTEN had no "Measurements", only "Thumbnails" – the spectra was literally stored (only) as the y-values of the thumbnail graph, with no x-axis, no metadata etc. So it was an improvement to move forward to "Measurements with Thumbnails". But there is a resource issue with EVERY Measurement having a ThumbnailWidget...what if we run a weekend collection taking 250,000 samples?
A better architecture might be just "Measurements" (the data), plus a ThumbnailBar able to dynamically generate and display ThumbnailWidgets on scroll events (similar to a Swift TableView, where cells are populated as they scroll into view, and are released when offscreen).
And possibly move away from the heavy "Widget" (with a couple Frames, Buttons, the pyqtgraph etc) to a simpler table or tree view which would probably use less memory.
For now, I'm compromising by treating the on-screen thumbnails as a ringbuffer, and kicking-off old Measurements (from memory, not disk) when we exceed a limit.
Note that the MeasurementFactory automatically saves new Measurements to disk (per SaveOptions) as they are created from Spectrometers, BEFORE they are handed to Measurements for addition to the Thumbnail bar. The process of creating a Measurement (from Spectrometer) and saving it to disk is atomic, so there is no need to worry that automatic deletion of Measurements for resource management will cause data to be lost.
It is ASSUMED that the user used the same Settings (integration time, boxcar smoothing, scan averaging etc) when taking the 'raw', 'dark' and 'reference' component spectra in generation of the 'processed' component. Only a single Settings object is retained for the entire ProcessedReading, which is copied when the Measurement is created.
That is, if the user does something like this, the wrong integration time would be stored with the Measurement:
There are various ways we could address this weakness (e.g., snap a Deep Copy of Settings on pause()), but it's not a priority at this time.
| enlighten.measurement.Measurement.Measurement.__init__ | ( | self, | |
| ctl = None, | |||
| processed_reading = None, | |||
| settings = None, | |||
| source_pathname = None, | |||
| timestamp = None, | |||
| spec = None, | |||
| measurement = None, | |||
| d = None ) |
There are three valid instantiation patterns:
| enlighten.measurement.Measurement.Measurement.add_pathname | ( | self, | |
| pathname ) |
| enlighten.measurement.Measurement.Measurement.build_row | ( | self, | |
| field ) |
Generate a single row of output for row-ordered CSV files.
The contents of the generated row depend on the 'field' parameter.
Metadata is only populated for x-axis fields and the FIRST ProcessedReading array.
| enlighten.measurement.Measurement.Measurement.clear | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.clone | ( | self | ) |
Called by PluginWidget.
| enlighten.measurement.Measurement.Measurement.csv_formatted | ( | self, | |
| roi, | |||
| prec, | |||
| array, | |||
| pixel, | |||
| obey_roi = False ) |
Used by save_csv_file_by_column and save_csv_file_by_row.
| enlighten.measurement.Measurement.Measurement.delete | ( | self, | |
| from_disk = False, | |||
| update_parent = False ) |
Release any resources associated with this Measurement.
Note that this will automatically delete the ThumbnailWidget from its parent layout (if emplaced), and mark the object tree for garbage collection within PySide/Qt.
| enlighten.measurement.Measurement.Measurement.display | ( | self | ) |
Display on the graph, if not already shown.
| enlighten.measurement.Measurement.Measurement.dump | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.expand_template | ( | self, | |
| template ) |
Some GUI text fields allow the user to enter strings containining "macro templates" which are dynamically expanded and evaluated at runtime.
Macros look like "{field_name}", where field_name can be any object attribute in wasatch.EEPROM, wasatch.SpectrometerState, or Measurement metadata (any field supported by Measurement.get_metadata). As a convenience some hardcoded macros are also supported, such as {time}.
Also I'd pull this out into a TemplateFeature.
| enlighten.measurement.Measurement.Measurement.generate_basename | ( | self | ) |
|
static |
generate a legacy Dash file header with one or more serial numbers
| enlighten.measurement.Measurement.Measurement.generate_id | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.generate_label | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.generate_today_dir | ( | self | ) |
| dict enlighten.measurement.Measurement.Measurement.get_all_metadata | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.get_csv_data | ( | self, | |
| pr ) |
Used by save_csv_file_by_column and save_csv_file_by_row.
| enlighten.measurement.Measurement.Measurement.get_extra_header_fields | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.get_metadata | ( | self, | |
| field ) |
This function is provided because legacy Dash and ENLIGHTEN saved row- ordered CSV files with very specific column headers and sequence which we don't want to casually break.
| enlighten.measurement.Measurement.Measurement.has_component | ( | self, | |
| component ) |
Not currently used.
| enlighten.measurement.Measurement.Measurement.id_callback | ( | self, | |
| declared_match ) |
Called (by way of ThumbnailWidget -> KnowItAll.Feature -> Measurements) when KnowItAll has generated a KnowItAll.DeclaredMatch for this Measurement.
| enlighten.measurement.Measurement.Measurement.init_from_dict | ( | self, | |
| d ) |
This is experimental.
Eventually we want to be able to load Measurements saved as JSON. We should also be able to receive externally-generated Measurements sent via JSON via the External API. This is reasonably key to both. Also it's handy to use this as a free-form intermediate format for parsing external formats like SPC.
| enlighten.measurement.Measurement.Measurement.interpolate | ( | self, | |
| new_settings ) |
Passed a SpectrometerSettings object (containing wavelengths, wavenumbers etc), return a copy of this Measurement's ProcessedReading which has been interpolated to the passed x-axis.
Interpolate on wavelength if available, otherwise wavenumbers, otherwise just copy the uninterpolated pixel data.
Who calls this?
| enlighten.measurement.Measurement.Measurement.is_displayed | ( | self | ) |
| enlighten.measurement.Measurement.Measurement.rename_files | ( | self | ) |
The measurement has been relabled (say, "cyclohexane").
So if pathnames contains "old.csv" and "old.xls", we want to rename them to "cyclohexane.csv" and "cyclohexane.xls". However, if those files already exist, we don't want to overwrite them (maybe the user is looking at a lot of cyclohexane). So if cyclohexane.csv already exists, just make cyclohexane-1.csv, etc. Whatever number we pick, apply it to all the pathnames (cyclohexane-1.xls, even if there wasn't already a cyclohexane.xls).
Also note that there are at least four ways this Measurement could have been instantiated:
| enlighten.measurement.Measurement.Measurement.replace_processed_reading | ( | self, | |
| pr ) |
We presumably loaded a measurement from disk, reprocessed it, and are now replacing the contents of the Measurement object with the reprocessed spectra, preparatory to re-saving (with a new timestamp and measurement_id).
| enlighten.measurement.Measurement.Measurement.save | ( | self, | |
| resave = False ) |
| enlighten.measurement.Measurement.Measurement.save_csv_file | ( | self, | |
| resave = False ) |
| enlighten.measurement.Measurement.Measurement.save_csv_file_by_column | ( | self, | |
| use_basename = False, | |||
| ext = "csv", | |||
| delim = ",", | |||
| include_header = True, | |||
| include_metadata = True, | |||
| resave = False ) |
Save the Measurement in a CSV file with the x-axis in one column, spectra in the next column and so on (similar layout as the Excel output).
Note that currently this is NOT writing UTF-8 / Unicode, although KIA- generated labels are Unicode. (Dieter doesn't seem to like Unicode CSV)
| enlighten.measurement.Measurement.Measurement.save_csv_file_by_row | ( | self, | |
| resave = False ) |
Save a spectrum in CSV format with the whole spectrum on one line, such that multiple acquisitions could be appended with one line per spectrum.
This is was the ONLY supported save format in Dash and legacy ENLIGHTEN, and still makes great sense for batch collections (it's much easier to append lines than columns to an existing file).
At the moment, this method is also being used to append new measurements to existing files.
Right now, we're using serial number in a new Measurement's measurement_id, hence label, hence filename. Having serial_number in a row-ordered CSV which aggregates multiple spectrometers is potentially confusing, but when we save the first spectrum we don't know that's the intention.
Note that currently this is NOT writing UTF-8 / Unicode, although KIA- generated labels are Unicode.
Note that Measurements saved while "appending" are NOT considered renamable at the file level, while Measurements saved to individual files are.
support .cropped, .interpolated
consider how to properly support verify_pathname and resave
| enlighten.measurement.Measurement.Measurement.save_dx_file | ( | self, | |
| use_basename = False, | |||
| resave = False ) |
| enlighten.measurement.Measurement.Measurement.save_excel_file | ( | self, | |
| resave = False ) |
Save the spectra in xls format (currently, one worksheet per x-axis)
As with save_csv_file_by_column(), currently disregarding SaveOptions selections of what x-axis and ProcessedReading fields to include, because there is little benefit to removing them from individual files. We can always add this later if requested.
Note that the data output is a little different from save_csv_file_by_column. This format probably should match that other format, but right now it doesn't.
A key difference is cropped (but not interpolated) ProcessedReadings. CSV files output every PHYSICAL pixel for most fields (pixel, wavelength, wavenumber, raw), and only substitute the "NA" for cropped values in "processed." Instead, this currently just outputs the cropped versions of everything.
| enlighten.measurement.Measurement.Measurement.save_json_file | ( | self, | |
| use_basename = False, | |||
| resave = False ) |
Save the Measurement in a JSON file for simplified programmatic parsing.
in the next column and so on (similar layout as the Excel output).
As with save_excel_file(), currently disregarding SaveOptions selections of what x-axis and ProcessedReading fields to include, because there is little benefit to removing them from individual files. We can always add this later if requested.
| enlighten.measurement.Measurement.Measurement.save_spc_file | ( | self, | |
| use_basename = False, | |||
| resave = False ) |
| enlighten.measurement.Measurement.Measurement.save_txt_file | ( | self, | |
| use_basename = False, | |||
| resave = False ) |
This is essentially the same as column-ordered CSV, but with no metadata, no header row and no commas.
(per proj "Pioneer")
| enlighten.measurement.Measurement.Measurement.to_dict | ( | self | ) |
Express the current Measurement as a single JSON-compatible dict.
Use this for both save_json_file and External.Feature.
| enlighten.measurement.Measurement.Measurement.to_json | ( | self | ) |
Render the current Measurement to JSON.
Use this for both save_json_file and External.Feature.
| enlighten.measurement.Measurement.Measurement.update_label | ( | self, | |
| label, | |||
| manual = False ) |
| enlighten.measurement.Measurement.Measurement.verify_pathname | ( | self, | |
| pathname, | |||
| resave = False ) |
| enlighten.measurement.Measurement.Measurement.write_processed_reading_lines | ( | self, | |
| csv_writer ) |
For row-ordered CSVs, output any ProcessedReading fields which have been selected.
In column-ordered files, it's not a big deal to always store dark / reference / raw, because they're easy to ignore if you're not using them. However, in row-ordered files, those extra lines can be really confusing, partly because of the "prefix columns" enforced by the file format (repeating wavecal coeffs etc), but also because when "appending" to existing files, it becomes very hard to tell which lines are spectrum-vs- component. So anyway, be careful not to print extra lines here that the user didn't request.
Regardless, there's no "caching" these across Measurements, because technically the user could have (and often would) take a new dark and reference repeatedly during a session. (They're not 'reasonably persistent' as with the x-axis.)
| enlighten.measurement.Measurement.Measurement.write_row | ( | self, | |
| csv_writer, | |||
| field ) |
| enlighten.measurement.Measurement.Measurement.write_x_axis_lines | ( | self, | |
| csv_writer ) |
For row-ordered CSVs, output any x-axis fields that have been selected.
Only do this if they have not yet been output for a given spectrometer.
To support multiple spectrometers, we're adding the spectrometer's serial number (as well as the x-axis unit) to the Notes field.
| bool enlighten.measurement.Measurement.Measurement.appending = False |
| enlighten.measurement.Measurement.Measurement.baseline_correction_algo = None |
| enlighten.measurement.Measurement.Measurement.basename = None |
|
static |
These appear in legacy saved spectra files as-written (Dash format), so don't casually screw with them (could break customer applications).
Note that all are scalars (no lists).
|
static |
| enlighten.measurement.Measurement.Measurement.ctl = ctl |
| enlighten.measurement.Measurement.Measurement.declared_match = None |
| int enlighten.measurement.Measurement.Measurement.declared_score = 0 |
|
static |
These fields weren't in the original Dash file format, so only use for the new "column-ordered" formats (including 'export')
|
static |
| enlighten.measurement.Measurement.Measurement.label = None |
| enlighten.measurement.Measurement.Measurement.measurement_id = None |
| dict enlighten.measurement.Measurement.Measurement.metadata = {} |
| str enlighten.measurement.Measurement.Measurement.note = "" |
| enlighten.measurement.Measurement.Measurement.pathname_by_ext = {} |
| str enlighten.measurement.Measurement.Measurement.plugin_name = "" |
| str enlighten.measurement.Measurement.Measurement.prefix = "" |
| enlighten.measurement.Measurement.Measurement.processed_reading = None |
| bool enlighten.measurement.Measurement.Measurement.renamed_manually = False |
| bool enlighten.measurement.Measurement.Measurement.roi_active = False |
|
static |
These CSV_HEADER_FIELDS only need to be used for row-ordered files.
| enlighten.measurement.Measurement.Measurement.settings = None |
| enlighten.measurement.Measurement.Measurement.source_pathname = None |
| enlighten.measurement.Measurement.Measurement.spec = None |
| str enlighten.measurement.Measurement.Measurement.suffix = "" |
| enlighten.measurement.Measurement.Measurement.technique = None |
| enlighten.measurement.Measurement.Measurement.thumbnail_widget = None |
| enlighten.measurement.Measurement.Measurement.timestamp = None |