aggregation-master
- aggregation_schema.json
- blog
  - experts.py
  - new_jungle.py
  - jungle_fourier.py
  - expected_information.py
  - old_weather
    - lines.py
    - db_setup.py
    - cell_extraction.py
    - MNIST_data
      - train-labels-idx1-ubyte.gz
      - t10k-labels-idx1-ubyte.gz
    - sampling.py
    - probabilities.py
    - corner.py
    - faces.py
    - cell_processing.py
    - image_tesseract.py
    - tess2.py
    - prob.py
    - digit_extraction.py
    - svm__.py
    - create_test_cases.py
    - __init__.py
    - corners.py
  - plankton.py
  - jungle.py
  - blanks.py
  - annotate_numbers.py
  - whales_2.py
  - exponential
    - _blankImage.py
    - _exponential.py
    - __init__.py
  - whales.py
  - sea.py
- .hound.yml
- LICENSE
- penguins
  - aggregate.py
  - mongo_to_postgres.py
- config
  - aggregation.yml
- engine
  - web_api.py
  - helper_functions.py
  - string_agglomeration.py
  - json_transcription.py
  - text_aggregation.py
  - local_aggregation.py
  - load_redis.py
  - classification.py
  - survey_aggregation.py
  - panoptes_ibcc.py
  - kdtree_multiclick_correct.py
  - latex_transcription.py
  - folger.py
  - job_runner.py
  - gorongosa_aggregation.py
  - spot_check.py
  - blob_clustering.py
  - heuristic_lcs.py
  - agglomerative.py
  - __init__.py
  - tests
    - basic_test.py
    - __init__.py
  - simplified_transcription.py
  - readme.md
  - rectangle_clustering.py
  - multiClickCorrect.py
  - csv_output.py
  - transcription_output.py
  - jobs.py
  - aggregation_api.py
  - clustering.py
  - text_clustering.py
  - annotate.py
- setup.py
- analysis
  - pulsar.py
  - folger_summary.py
  - old_weather2.py
  - thorough_transcription.py
  - transcription_analysis.py
  - folger_summary2.py
  - old_weather.py
  - heatmap.py
  - __init__.py
  - vic.py
  - folger_analysis__.py
  - folger_analysis.py
  - whales.py
  - annotate_analysis.py
- experimental
  - chicago
    - aggregation.py
    - __init__.py
  - cancer
    - gold.py
    - __init__.py
    - cancer.py
  - mongo.py
  - soybot
    - compareGold.py
    - createGold.py
    - __init__.py
  - consensus
    - consistency.py
    - __init__.py
    - paramSearch.py
  - pulsar.py
  - milkway
    - dbscan.py
    - geometry.py
    - __init__.py
    - collectGoldStandard.py
  - graphicalClassification
    - MajorityVote.py
    - attributeBased
      - majorityVote
        MajorityVote.py
        UserNode.py
        __init__.py
        SubjectNode.py
      - __init__.py
    - MultiClassMajorityVote.py
    - MultiClassSubjectNode.py
    - __init__.py
    - BaseNodes.py
  - setup
    - serengetiPermutation.py
    - __init__.py
  - lhc
    - dbscan.py
    - __init__.py
  - kelp
    - kelp.py
    - __init__.py
  - classification
    - Container.py
    - IBCCserengeti.py
    - IBCCcontainer.py
    - UserContainer.py
    - SubjectContainer.py
    - __init__.py
    - pyIBCC.py
  - galaxyZoo
    - analyze2.py
    - trim.py
    - blankImage.py
    - analyze.py
    - __init__.py
    - analyze3.py
    - simulation.py
    - galaxyIBCC.py
    - positiveExamples
  - roc_fun.py
  - penguins
    - newCluster.py
    - dbscan2.py
    - distanceAnalysis
      - closestPenguin.py
      - findImage.py
      - above_or_below_average.py
      - __init__.py
      - distance.py
    - cython
      - 5_10_15.py
      - divisiveDBSCAN.py
      - 5_10_15.pyx
      - setup.py
      - helloworld.pyx
      - __init__.py
      - helloworld.c
      - 5_10_15.c
    - goldStandard.py
    - bias.py
    - dbscan.py
    - dkMeans.py
    - mongoDBSCAN.py
    - clusterAnalysis
      - errorCheck.py
      - 5_10_15.py
      - weightedMajorityVote.py
      - sampler.py
      - penguin_ibcc.py
      - goldStandard.py
      - ali_plot.py
      - check4.py
      - anomalyDetection.py
      - errorCheck2.py
      - completeAnnotation.py
      - check3.py
      - roc.py
      - confusionMatrix.py
      - penguins_at_5.py
      - dDBSCAN.py
      - DBSCAN.py
      - stats.py
      - check2.py
      - check.py
      - ibcc_priors.py
      - plot4.py
      - rangeSearch.py
      - plot5.py
      - __init__.py
      - plot3.py
      - votingAnalysis.py
      - blankPhotos.py
      - plot6.py
      - ratio.py
      - plot2.py
      - plot1.py
      - distance_.py
    - merge_goldstandard.py
    - __init__.py
    - contest.py
    - cluster.py
    - visual.py
    - blank.py
    - goldStandard_chick.py
    - blur.py
    - variance.py
    - hello.pyx
  - algorithms
    - t.py
    - list_panoptes_projects.py
    - hashdump.py
    - limited_ibcc.py
    - ouroboros_api.py
    - migration.py
    - chimp.py
    - blanks
      - serengeti3.py
      - serengeti2.py
      - image_diff.py
      - wildebeest.py
      - user_analysis.py
      - cloudsight.py
      - cloudsight2.py
      - serengeti.py
      - wildebeest2.py
      - serenget4.py
      - __init__.py
      - serengeti5.py
    - penguin.py
    - clustering_dict.py
    - ellipse_approximation.py
    - cass.py
    - test.json
    - wildebeest.py
    - jungle3.py
    - penguin_analysis.py
    - serengeti_blank
      - serengeti_cass2.py
      - __init__.py
      - serengeti_cass.py
    - folger.py
    - old_weather
      - t.py
      - alignment.py
      - rotation.py
      - nearest_neighbours.py
      - create_data.py
      - test_data.py
      - bear.py
      - cells.py
      - scaling.py
      - loader.py
      - test.py
      - neural_network.py
      - current_oldweather.py
    - blanks.py
    - serengeti.py
    - __init__.py
    - user_quality.py
    - serengeti_2.py
    - solar_watch.py
    - cluster_count.py
    - cell.py
    - automatic_optics.py
    - divisive_kmeans.py
    - test.py
    - csv_output.py
    - jungle2.py
    - marmot.py
    - retrieveScript.py
    - quad_tree_correction.py
    - new_agg.py
  - clusteringAlg
    - fix.py
    - clusterCompare.py
    - flatFix.py
    - adaptiveDBSCAN.py
    - decay.py
    - divisiveDBSCAN.py
    - ibccCorrect.py
    - __init__.py
    - divisiveKmeans.py
    - divisiveKmeans_2.py
    - multiClickCorrect.py
    - zeroFix.py
    - divisiveDBSCAN_multi.py
  - clustering
    - __init__.py
    - dbscan
      - sample.py
      - plot.py
      - scikit.py
      - __init__.py
      - kmeans.py
    - friendsOfFriends
      - reducer1.py
      - mapper1.py
      - reducer.py
      - __init__.py
      - mapper.py
      - dbscan
        __init__.py
  - scala
    - src
      - aggregation
        SimpleApp.scala
    - .settings
      - org.scala-ide.sdt.core.prefs
    - target
      - .history
    - .gitignore
    - .classpath
  - paper
    - errorCheck.py
    - checkIBCC.py
    - penguinScript2.py
    - condorAggregation.py
    - confidence
      - penguinBlank.py
      - confidence.py
      - __init__.py
      - blankConfidence.py
    - condors.py
    - confidenceAnalysis.py
    - camload.py
    - analyze.py
    - analyzeChange.py
    - camrun.py
    - condorScript1.py
    - clustering
      - penguinAgglomerative.py
      - clusterCompare.py
      - penguinGoldStandard.py
      - penguinAggregation.py
      - aggregation.py
      - clusterCompare2.py
      - __init__.py
    - setup.py
    - multiClick.py
    - __init__.py
    - soyman.py
    - chicksEggs.py
    - penguinScript1.py
    - clustering.py
    - nonEmptySerengeti.py
    - downloadCam.py
  - old
    - quadTree.py
    - IBCCmultiClass.py
    - nonEuclideanKMeans.py
    - IBCCsetup.py
    - IBCCmongo.py
    - kMedoids.py
    - logisticRegression.py
    - __init__.py
    - kMeans.py
    - IBCCmajorityVote.py
    - serenegtiExpertConsensus.py
    - IBCCanalyze.py
  - mongo
    - galaxyStats.py
    - __init__.py
    - IBCC.py
  - spacecells
    - cells.py
    - __init__.py
  - chimp
    - __init__.py
  - classifier
    - majorityVote.py
    - graphicalEM.py
    - iterativeEM.py
    - __init__.py
    - graphicalIBCC.py
  - imageCompare
    - analyze.py
    - image.py
    - timeSeries2.py
    - timeSeries.py
    - __init__.py
    - image2.py
    - sun.py
  - condor
    - experience.py
    - DBSCAN2.py
    - animal_IBCC2.py
    - initial.py
    - site.py
    - goldStandard.py
    - animal_EM.py
    - animal_EM2.py
    - mapReduce
      - __init__.py
    - condorIBCC.py
    - correlation.py
    - retireBlank.py
    - site2.py
    - animal_IBCC.py
    - check_IBCC.py
    - error_plot.py
    - animal_EM3.py
    - paper
      - agglomerativeDivisive.py
      - __init__.py
    - DBSCAN.py
    - presentation
      - ibccDK_Means.py
      - independence.py
      - plurality.py
      - roc.py
      - gold.py
      - __init__.py
      - condor_IBCC.py
    - analyzeBlank.py
    - goldStandardBlank.py
    - __init__.py
    - blankIBCC.py
    - condorIBCC_3.py
    - double_entry.py
    - cluster.py
    - gain_per_user.py
    - temp.py
    - snapshotApproach.py
    - condor_or_no_condor.py
    - error_examine.py
    - completionStats.py
    - simulation.py
    - test_error_correction.py
    - divisiveClustering.py
    - confusion.py
    - blank.py
    - condor_EM.py
    - aws
      - script
      - config.py
      - __init__.py
    - condorIBCC_2.py
    - find_site.py
  - __init__.py
  - hive
    - majorityVote
      - f.py
      - MVreducer.py
      - reducer1.py
      - mapper1.py
      - reducer.py
      - __init__.py
      - mapper.py
    - __init__.py
    - weightedVote
      - voteDistribution.py
      - consistencyReducer.py
      - filter.py
      - weightedClassification.py
      - __init__.py
      - classificationDistribution.py
  - plankton
    - quick.py
    - quick2.py
    - incomplete.py
    - repeats.py
    - missing.py
    - .Rhistory
    - __init__.py
    - cluster.py
  - stats
    - __init__.py
    - serengetiIndependence.py
    - serengetiExpert.py
  - transcribe
    - longestCommonSubsequence.py
    - __init__.py
  - dkMeansPaper
    - dk1.py
    - __init__.py
  - mapreduce
    - condor
      - __init__.py
    - __init__.py
    - reducer
      - DBSCAN.py
      - __init__.py
    - serengeti
      - reducer.py
      - __init__.py
      - mapper.py
    - mapper
      - __init__.py
  - serengeti
    - time_series.py
    - individualIBCC.py
    - newblob.py
    - blob4.py
    - sift.py
    - blankImage.py
    - blob.py
    - analyze_marten.py
    - SnapshotSerengeti.py
    - attributeIBCC.py
    - bias.py
    - analyze_marten2.py
    - sessions.py
    - independence.py
    - blob3.py
    - estimateConfusion.py
    - attributeIBCC2.py
    - blob2.py
    - attributeMajorityVote.py
    - blanks.py
    - filter.py
    - edgeDetector.py
    - IAAI
      - weight.py
      - pearson.py
      - alg2.py
      - alg4.py
      - weight2.py
      - betaTest3.py
      - alg5.py
      - majorityVote.py
      - nodes.py
      - betaTest.py
      - confusionMatrixConsistency.py
      - currentAlg.py
      - betaTest2.py
      - temp
      - numberConsistency.py
      - subjectStats.py
      - tauTest.py
      - alg.py
      - subjectStats2.py
      - subjectStats3.py
      - currentCorrect.py
      - alg3.py
      - confirm.py
      - blankPhotos.py
      - test.py
      - timeSeriesAnalysis.py
      - bySpecies.py
      - confusionMatrixConsistency2.py
    - blankTwo.py
    - __init__.py
    - serengetiSplit.py
    - confusion.py
    - aws
      - script
      - config.py
      - __init__.py
  - aws
    - aggregation_engine.py
    - __init__.py
    - mapreduce
      - param.py
      - __init__.py
      - mapper.py
      - blank.py
    - serengeti
      - majorityVote
        reducer.py
        __init__.py
        mapper.py
      - __init__.py
- .travis.yml
- README.md
- active_weather
  - tessdata
  - sliding2.py
  - text_detection.py
  - eval.py
  - mnist_based_classification.py
  - MNIST_data
    - train-labels-idx1-ubyte.gz
    - t10k-labels-idx1-ubyte.gz
  - init_analysis.py
  - sobel_transform.py
  - old
    - tess.py
    - create_template.py
    - classification.py
    - update.py
    - otsu.py
    - input_data.py
    - prob_hough.py
    - unicharset
    - paper_interpolation.py
    - paper_sobel.py
    - tesseract_font.py
    - paper_otsu.py
    - hocr.py
    - plot_grid.py
    - gold_standard.py
    - backup_weather.py
    - paper_baseline.py
    - active_weather.py
    - tess_eval.py
    - __init__.py
    - temp.py
    - test.py
    - template_align.py
    - database_connection.py
    - learning.py
    - paper_threshold.py
    - extract_digits.py
    - tense_weather.py
    - pca.py
    - gradients.py
    - align_images.py
    - reduction.py
    - paper_houghes.py
  - tesseract_update.py
  - active_weather_backup.py
  - example.py
  - learning2.py
  - active_weather.py
  - experiment.py
  - __init__.py
  - pca_example.py
  - paper_quad.py
  - segmentation.py
  - learning.py
  - tensor.py
  - sliding_window.py
- start.sh
- requirements.txt
- vision
  - __init__.py
  - butterflies.py
- .eggs
  - README.txt
- .idea
  - inspectionProfiles
    - profiles_settings.xml
    - Project_Default.xml
  - modules.xml
  - encodings.xml
  - .name
  - scopes
    - scope_settings.xml
  - vcs.xml
- Dockerfile
- .gitignore
- Stargazing_2015
  - bayesian.py
  - analysis.py
  - user_scrap.py
  - crontab
  - scrap.py
  - iterative_aggregation.py
  - distribution.py
  - aggregation.py
  - stats.py
  - postmortem.py
  - postgres_aggregation.py
  - diff.py
  - meta.py
  - loadtesting.py
  - __init__.py
  - Dockerfile
- docs
  - Makefile
  - source
    - blob_aggregation.rst
    - ouroboros.rst
    - images
      - rectangle_graph.py
      - __init__.py
      - butterfly1.jpeg
      - rectangle_overlap.py
    - text_aggregation.rst
    - examples
      - json_aggregation.py
      - penguin.py
      - __init__.py
    - engine.rst
    - blanks.rst
    - aggregation.rst
    - engine.tests.rst
    - getting_started.rst
    - ocr.rst
    - marking.rst
    - json.rst
    - point_aggregation.rst
    - machine_learning.rst
    - modules.rst
    - error_handling.rst
    - index.rst
    - conf.py
    - graphics.rst
    - aggregation_results.rst
    - rectangle_aggregation.rst
    - standalone.rst
    - classifications.rst
    - output.rst
  - make.bat
- supervisord.conf
- time.py
- docker-compose.yml

Zooniverse Aggregation Code

This repo allows you to do aggregation with Zooniverse projects. Aggregation is the process of taking classifications/markings/transcriptions from the multiple users who see each subject and combining them into a "final" answer that is useful for researchers. For example, if 3 out of 4 people say that an image contains a zebra, aggregation would report that there is a 75% probability that the image does indeed contain a zebra.

Documentation for the Zooniverse Aggregation code can be found at https://developer.zooniverse.org/projects/aggregation/en/latest/ (We are currently experiencing severe problems with loading (lots of 403 errors and pages just not loading). We've raised issue and are looking into alternative hosting options - but in the mean time just keep reloading until you get the site to show.)

There is a draft of a paper that I was planning on writing on ocr and Tesseract. There is lots of random code in the active-weather directory and a link to an unfinished paper I was working on - that paper has some good ideas on how to do line detection/extraction for tabular data.

The directory to do all of this in "engine". This is the code base that runs every time you press "export aggregations" in the builder builder page. You can also run things locally if you want - this is especially useful if you have an Ourboros project (just ignore that if you don't already know what Ourboros is) or if you want to do bespoke aggregation or fix a bug.

Running the Aggregation Code Locally

The aggregation engine needs a bunch of stuff installed and running such as PostgreSQL and Cassandra. While you can manually install all of the required packages and set them up, it is much easier to use Docker. Use the following steps to get the aggregation docker instance running on your computer.

If you don't have Docker installed and running on your computer, go to the site first and follow the instructions.
Clone the aggregation github repo on your computer, i.e. git clone git@github.com:zooniverse/aggregation.git
In your cloned instance of the aggregation repo, run docker-compose up. This may take a while the first time you run it.

You should now have several docker containers running. You can check with docker ps. You'll need a copy of the Panoptes database running which will give you the subject classifications/markings/transcriptions for all the projects. Talk to Adam or Cam about where to get them from - you'll get a file like "panoptes-panoptes-production-2016-04-25-04-56.dump". One of the containers that is running the PostgreSQL container. You'll need to put that file into the PostgreSQL container and run pg_restore to restore the image.

In the aggregation repo directory, there should now be a sub-directory called "data". Copy/move the .dump file into this directory. The data subdirectory is presistent - you'll only need to do this once and the database will continue to exist even after you rebuild any code. (Of course from time to time you may need to download an updated copy of the dump file.)
Connect to the postgres container with docker exec -it aggregation_postgres_1 bash
Now you'll need to create the panoptes database (this database will be empty until you run pg_restore) with the command createdb -U postgres panoptes_development (run this in the terminal not in psql)
Before restoring, you'll need to set a few environment variables
- db='panoptes_development'
- username='postgres'
- local_backup_file="panoptes_staging.dump" (change this to be whatever the specific name of the .dump file you have is)
Finally run pg_restore --clean --verbose -Fc -h localhost -U ${username} -d ${db} ${local_backup_file}. This will take a while (but again you only need to do this once)

You'll need a file called aggregation.yml to provide a bunch of setup values. A simple example file is in config/aggregation.yml in the aggregation repo and is already loaded for you into the docker image. In production, this YAML file allows us to make a secure (admin) connection to Panoptes as well as Rollbar (for reporting errors) and connecting to the postgres/cassandra db. For development, you just need to connect to the postgres/cassandra dbs in the docker images and the default values in the aggregation.yml file will allow you to do that. You only need a secure (possibly) admin connection to Panoptes to do a few specific things (such as for Annotate/Shakespeare's world being able to retire subjects via an API call). For the "development" environment in the YAML file we just use a public connection to Panoptes (which I don't think even really gets used). The YAML file used in production (ask Adam for details/location) has all the necessary password/tokens etc. to make all the necessary connections.

There is one password that you will need to set in the YAML file and that is the postgres password. The docker image comes with the default password which could probably just be included in the YAML file but to play it safe is not included. If you want to set the postgres user password, log into the postgres container and in psql use "alter user postgres password 'apassword';"

The aggregation engine is now ready to be run. Exit the postgres container and use the following steps

docker exec -it aggregation_aggregation_1 bash
cd engine
./aggregation_api.py project_id development
The first time you run it, make sure that __reset_cassandra_dbs__() is executed, i.e. right after project.__setup__(), run project.__reset_cassandra_dbs__(). This only needs to be done once

project_id is the numerical identifier for a project. You can search for the number using lita (on Slack) with something like lita project wildcam which will tell the project IDs of all projects which have "wildcam" in their title. Assuming that everything worked - the aggregation_api will save the results to the /tmp directory in the Docker image (no email will be sent out). There will be both a directory of results with your project id and tar.gz file. You can use docker cp to extract the results to your local directory.

Troubleshooting

Sometimes something goes wrong and the aggregation engine crashes. There are 3 reasons why this could happen.

A system crash - for example AWS goes down (but hopefully not). This is probably the rarest cause and Adam/Cam/etc. should probably realize that something is up with AWS
Newly deployed code interrupts an aggregation run - when deploying new code, there is no way of checking to see if there is currently an aggregation run in progress. Any such runs will automatically be stopped. The file spot_check.py runs every hours to see if there are any interrupted runs. If so an email is sent out to Zooniverse admin (the email is specified in the aggregation.yml file) (There is currently no way to automatically restart any such runs but this is again a relatively rare case.)
Edge cases - something about your classifications causes an expected case in the aggregation code and Python raises an exception. This is by far the most likely cause of error. There is currently no way of automatically emailing people to let them know that such as a case has occurred (definite future plans) but any such errors are reported automatically via Rollbar and github. To see the errors in Rollbar you need to have an account there and access to the Zooniverse_Aggregation dashboard. Currently Cam, Adam, Marten and Greg have such access. Rollbar will also automatically raise an issue on github (in the Aggregation repo) and show the Python exception. However, currently the Github issues do not report the project id so you may not be completely sure if your project caused a specific error. (The project ids are reported in Rollbar).

Aggregation runs can take a little while - depending on how big your project is and what type of tools you used (polygon aggregation takes longer than simple classification aggregation). Depending on the current load on Panoptes, it might take a little while for the aggregation to start. Currently there is no way of checking to see the current state of the aggregation. If you haven't received an email with a link to your results in a couple of hours, check github to see if any new issues have been raised. In the future we aim to send out emails saying when an aggregation run has crashed.

Using Classification CSV files as input

You can also use the classification csv files (available from the panoptes project builder page) as input for the aggregation engine. If you use the CSV files you do not need to connect to postgres or cassandra. However the current implementation is slightly inefficient since values are stored in a Pandas dataframe without indexing (this could definitely be improved in the future). You will need to also have a YAML file located in /app/config/aggregation.yml with the following code:

development: &default
  panoptes_username: TBD
  panoptes_password: TBD

Since the aggregation code will need to connec to Panoptes to get some information. (The location could be updated in the future - /app/config is just where the file is on AWS) The code to run aggregation locally is in local_aggregation. The code to run it is:

from local_aggregation import LocalAggregationAPI
project = LocalAggregationAPI(63,"/home/ggdhines/Downloads/copy-of-kitteh-zoo-subjects-classifications.csv")
project.__setup__()
project.__aggregate__()

with csv_output.CsvOut(project) as c:
    c.__write_out__()

Important Notes

The aggregation code for Annotate and Shakespeare's world runs daily but sends out a weekly summary email to the project team. This email is sent out on Tuesdays (nothing special - just first set it up on a Tuesday). The code takes a while to run so probably best not to deploy new code on Tuesdays.

You can also log on to the EC2 instance and check to see if the processes are running. Either ps -auxf or top -c will work. top -c might be a bit better since the aggregation code for AnnoTate and Shakespeare's World takes up a lot of CPU so they tend to be at the top of the list.