Incorporated improvements made for suh_pipelines with respect to job batch processing and tolerance of normalization errors
Changes to suh_pipelines for web update
Relies on Stephen’s Google Drive for demo experiments
1) Generalize web update code that was previously non general and specific to AutoGate.
2) New GUI support for when Google Drive is used and files are TOO big for automatic direct download.
3) Use WebDownload module in case Amerige server is down and the fcs samples are found on Stephen Meehan’s Google Drive account.
1) Shows brings in UMAP v3.01 improvements to DimensionStacker and PredictionAdjudicator.
2) Generalize web update code that was previously non general and specific to AutoGate
Incorporates the improvements to PredictionAdjudicator made for UMAP v.301.
1) 1)New SuhWindow manager module which supports smoother window movement between follower and followed windows.
2) Also provide distinct windows for training/test set when matching
3) Support easier API for putting java component in tooltip
4) IF asking tooltip x and y offsets to be ‘half’ then adjust for both component and size of tip.
5) Fix exception caused by new functionality
1) Multiple ROI (regions of interest … aka manual gates) can be on UMAP plot.
2) user can play with DBM details for clustering 2D UMAP islands
3) Polygon ROI can be computed from either a DBM cluster on a UMAP island or on the 94% data points closest to the median of a population associated with a prior classification label)
4) All UMAP ROI objects (manual gates) can be given a name and saved in a file for export to FlowJo.
5) The KLD table synchronizing can be more easily cancelled and turned on
1) Incorporate new improvements in UMAP and EPP when user plays with HierarchyExplorer.
2) Fixed a couple of exceptions introduced
1) Make companion windows less JUMPY and less prone to being hidden (e.g. KLD + measurement distribution window).
2) Clean up menu language
3) Improve GUI for saving polygon gates in UMAP/UST for selected subset labels . Allow user to save polygons and their label names
4) Rename MatBasics modules for converting to/from polygon data
Give user of run_umap a checkbox on the toolbar to silence the automatic synchronizing of ROI objects to the table of measurement distributions and KLD
Avoids exception when using run_umap with new fast_approximation feature.
1)fixes data “edge case” in modal EPP where an infinite loop was occuring in the omip-069 data set when doing 39 stains on the CD45+ gate with W=.01
2)Fixes older problems recently discovered when doing “quick and dirty” UMAP on multiple merged samples
Avoids exception when opening ReagentTable while PlotEditor is focussed on UMAP output or tSNE output.
Contains fixes to DimensionExplorer in UMAP if called from EPP HierarchyExplorer
Possible fix to bug of launching AutoGate2020a.app from subfolder of AutoGate.app
Addresses compiler only issue with EPP HierarchyExplorer window
1) Incorporates many cosmetic improvements needed for v2 of my EPP findings paper. Similarity and F-measure historgrams are more clear and consistent.
The running of run_epp is much nicer and allows access to UMAP for the same data run on by EPP.
1)Defend against exception in denormalize function if EPP run from AutoGate produces NOTHING !!!
2)Call quarantine of Wayne’s modal clustering mex at startup of AutoGate
3)Update SuhModalSplitter.Quarantine function to address issue caused by reorganization of EPP for open source distribution to MathWorks File Exchange
4)Improvement to Epp Hierarchy Explorer GUI so that the parent/chiild gate plots in the uitree are consistent with the idiom established by AutoGate
Reformat question about exit
1) New “EPP hierarchy explorer” comes up by default at end of EPP run to supply very quick tree viewing of EPP results and the KLD table with measurement spreads and X/Y plot of EPP split and X/Y plot of any selected rows in KLD table.
2) ReagentTable now has the ability to see an X/Y plot preview for any 2 row selections … copied behavior from Kld table.
3) For run_epp when doing match table (QF table) fow selections acvitate the Kld table for the associated picks
Enhances ability to inspect underlying data rows for KLD table by;
a)incorporating the PersonalizableTableMode MeehanMetaSpace developed from Sept 2002 until Jan 2013 and using it to inspect each row beng summarized in KLD table
b)allowing export to Excel and launch.
Prevents user from creating mroe than one EPP hierarchy under a set of top EPP gates
Fixes issue where when user has no cells selected in PlotEditor then the ReagentTable will export csv file for parent gate’s cells
Introduces feature for saving individual events to a csv file from the toolbar of the KLD table (also ReagentTable)
Provides fix for exception due to bug introduced on Aug 23 by me when working on computing polygons from DBM clustering
Fixes another edge case in the 3rd new EPP splitting method (polygons via DBM clustering
Fixes issues found testing with new computer and using new EPP split method of producing polygons via DBM clusters
1) A new 3rd mode for EPP whereby it produces polygons based on DBM clustering
2)Much faster updating of GatingTree when producing polygons from EPP
1)Fixes typo bug that made modal EPP unusable in previous version
2)Enables making polygons from cluster picks quickly
Improvements to copy quality
1)Cosmetic cleanup to EPP config screen
2)Use Wayne’s July 31 2021 github update with fix to assert problem that deprecates my inferior workaround hack
Makes Modal EPP the default (not the old DBM EPP)
1)Exception with empty polygons when NOT doing best balance
2)Issue where polygons are splitting projections that have a single DBM cluster with high density bandwidth
Reorganizes SuhEpp code so that a run_epp executable is possible
Gets the new EPP modal clustering running on R2020a
Introduces first version of Wayne’s new EPP that uses modal clustering instead of DBM clustering
1) C++ speedup for Separatrix and cluster border
2) Integration with cloud apps – Google Calendar
3) Integration with cloud apps – Google Tasks
Bug fixes to EPP
1) Continued improvements to config screens for EPP, MDS, Hi-D match and Phenograms: easy access to publciations/slides, more compact, smooth and easy to read.
for more details on improvements http://cgworkspace.cytogenie.org/Tutorials/ThingsInV4634.html
2) Provide access to EPP slideshow on EPP
3) Fix wordings for EPP, multidimensional and Hi-D.
Attempts parallelism in the JAVA implementation of Separatrix combinatorics.
Expands EPP to
1) Measure a balanced split using BOTH clustered and unclustered cell/event counts. Previously a balanced split only counted cells/events in a cluster.
2) Include unclustered cells/events when testing criteria of minimum size of leaf gate IF the user DE-selects BOTH the checkboxes âTrim leaf gates?â and âApply to branches?â. Previously EPP only counted clustered cells when
3) Provide GUI support for both the above EPP improvements to the EPP configuration screen and the PlotEditor configuration screens for seeing separatrix and 2D PathFinder.
4) Allow user to ignore reuse of prior cached score with new checkbox on 2D PathFinder configuration screen
Upgrades 2D PathFinder for Separtrix so that it obeys the user settings for :
a) X/Y reuse according to KLD and DULL levels.
b) minimum size of a leaf
Provides FIRST full integration of the FIRST step on the long many-step road to having a full fast parallelistic EPP rewrite. The JAVA acceleration is now available as an optional alternative to the stable MATLAB so that users can help test. Next the JAVA accleration will get threads for 8 clusters or more.
The reason the full EPP rewrite is going to JAVA first is that it is an easier transition from MATLAB to C++ than going directly to C++. Java is very similar to C++ but does not freeze the computer.
So moving from vectorized, weak-typed language to a non-vectorized and strong-typed language is quicker and more stable by going to JAVA first.
1) Control disk storage on CytoGenie.org for AGI files by # of gigabytes. Wayne, Brissy and Stephen get 10GB. AutoGate runtime get 1GB. MATLAB full get 2GB.
2) Provide a mechanism for preventing the running out of disk space by stopping the addition of new users by setting up a file admin.properties in the folder /usr/local/tomcat7/webapps/ROOT/GetDown2/AutoGate/ExperimentFiles with the property setting AllowNewUsers=true.
1) New feature allowing user to save to CytoGenie.org server but have a limited # of files and use an rsa key that is contained in an encrypted zip file and then deleted IMMEDIATELY after being used
2) Improve the work flow for importing an agi file either from the main window or GatingTree window so that user does not have copy the agi file to the correct experiment location.
3) By default limit the # of cells display to speed up (sigh) the display of Cytek Data
1) Fix problem with r2019 and later when drawing ellipse on a non logicle scale
2) Improve htmo from MDS window
addresses EPP issues found with new setting features
1) Out of memory side effect when resuming EPP with max # clusters>=5
2) Edge case when small clusters all have the same X or Y value and cluster outliers are selected
3) Support 1D KLD avoidance with uncompensated data
More fussy programming to the Separatrix debug screen which the PlotEditor laucnhes to explain these suspicious plots with things like hair pin borders.
a) New google apps/drive
b) New user default settings for Wayne’s new EPP “tuning knobs” (max # clusters, min applies to branch and naming stuff)
1) The majority of UMAP v2.2 benefits
2) 3 new functions for EPP as clarified by Wayne in the last 2 weeks
a. Gate names do not mention a marker if it does no contribute to the split
b. A form of dynamic bandwidth is afforded by having a maximum # of ciusters. See red box on RIGHT sight of screenshot below. When this setting is surpassed EPP will shift down the level of detail until either the # of clusters is less than the max … or until the cluster detail level of very low is reached
c. The minimum cells setting now can apply to either branch or leaf. Before it only applied to the leaf. See red box on LEFT sight of screenshot below.
Improves the PlotEditor’s separatrix comparison debugger so that it is easier to read and also removes a bug where the best green line gets redrawn for EVERY case. No changes to separatrix logic or EPP was necessary since the improvements confirmed it is working
1) Activate UMAP Dimension Explorer and plot flashing when selecting in QFTable either in the context of run_umap or AutoGate Parameter Reduction window
2) Add subset name to toolbar of ReagentTable and UMAP Dimension Explorer
Mostly cosmetic and wording improvements in support of the 1st draft of the Cytek tutorial for EPP on T cells
Introduces simpler GUI for selecting AutoGate demo experiments and reading the associated documentation.
Fixes sizing and coloring for the 1D data distribution pseudocolor bar in the ReagentTable and Umap Dimension Explorer.
Continues improving KLD ReagentTable and clarifying wording with gooder English
1)Fix main problems with MATLAB uitable: fast row wide selection as opposed to TERRIBLY SLOW cell selection, right alignment of numbers, correct sortng of negative numbers, a few other small things.
2)Move Kld table’s generalized logic on its toolbar to the toolbar super class TablePicker for WHEN EVER modal is FALSE a) Always refresh and refresh logic b) Screen capture and browse logic c) Size info (rows x columns) report
3)New features for run_umap like dimension explorer with region of interest tools
1) Adhere to David Parks to show DULL numbers so that low is dull and high is bright.
2) Add tool tips to the ReagentTable
3) Improving wording and descriptions about KLD, our dull SDU metric, all columns of the ReagentTable, the tool tips for “Picking reagents” etc.
4) Make the Epp form more clear and consistent.
Provides better descriptions and wording for reagents and KLD.
Continues to improve on reagent picker logic and data sample file creation with excel etc.
1)Continue improving the look, behavior and consistency of reagent picking from GatingTree, table or list in all invocation scenarios.
2)For consistency with modal and mode less window/figure ALWAYS put the All checkbox at bottom left since MATLAB forbids any top toolbar or stuff when style==modal
Fix continued parade of problems associated with toggling between modal JAVA windows for picklists like ReagentTable or GatingTree which are modal MATLAB figure windows. Weird event issues. Only Epp window is simple since it is a non modal mode-less JAVA window and does not fight with MATLAB figure conventions.
necessary to provide
1) MAJOR bug discovery (Feb 24, 2021) and fix (Feb 25, 2021) to the NEW QFMatch optimization by overlap which was invented in October 2020
2) Several cosmetic fixes due to working on high def MS Windows computer
Provide consistent GUI (btns etc) for picking parameters from 3 sources (GatingTree, ReagentTable or popup list) on Epp dialog, MDS dialog and File saving (csv, fcs)
1) Accelerates CD sorting
2) Allows user to see 1D path finder in FCS parameter pick list
3) Shows applied and model gate images QUICKLY from GateCopyQualityDashboard
1) Add 1D PathFinder bars as html text to FCS parameter pick list when computing KLD
2) Express matching positively as similarity instead of dissimilarity
3) Show Wayne’s picture for false positive/negative
4) Spell QFMatch consistently like publication
5) Put in background reading combo box for reading publications on QFMatch, EPP/MDS and EMD
6) When going through GateCopyQuality dashboard show both applied and model gate images in tooltip or in popup window for fast convenience.
1) Allow user to pick EPP parameters from list using sorted KLD
2) Cosmetic improvements.
Adds KLD to fcs parameter picklist.
Contains improvements to copy quality view of gates for mass+distance similarity checking.
1) Hides QF match choice when applying/copying gating sequences
2) Maximizes UMAP speed by scaling parallelism by the # of logical CPU cores the OS assigns MATLAB
Includes speedup to stochastic gradient descent with UMAP
1) Support GLOBAL colors by name object to improve synchronization of updates and changes…. next improvements will include the JAVA listener model so that all ColorsEditor windows HEAR changes to global defaults… YAY.
2) A major consequence of 1) is that fnTranslate logic can NOT be global since different GatingTree’s have different names for the same ID that needs translation. Instead all transation needs have to be parameters to the methods in ColorsByName which need to do translation
3) Fix problems associated with having treated this.dflts the same way REGARDLESS of instance type: ColorsByName or java.util.Properties. This effectively meant any updates are not seen by users of ColorsByName and it meant that lookup of colors was case senstive instead of case insensitive
1) Consolidation of re-factoring of cluster matching AS WELL AS cluster tethering of manual gates During 2D gate sequence building. Ready for extensive white box testing with OMIP 069, Eliver’s B-cell/Macrophage and Nicolay’s panoramic data set.
2) Fix bug with getMatchName argument counting