Project: kipoiseq (GitHub Link)

kipoiseq-master
- .circleci
  - gitconfig
  - deploy_docs.bash
  - config.yml
- Makefile
- kipoiseq
  - dataloaders
    - splicing.py
    - __init__.py
    - protein.py
    - sequence.py
  - extractors
    - gtf.py
    - fasta.py
    - vcf_query.py
    - vcf.py
    - __init__.py
    - protein.py
    - vcf_seq.py
    - vcf_matching.py
    - multi_interval.py
    - base.py
  - __init__.py
  - utils.py
  - transforms
    - functional.py
    - __init__.py
    - transforms.py
  - dataclasses.py
- LICENSE
- CONTRIBUTING.md
- notebooks
  - getting-started.ipynb
  - getting-started-with-VariantExtractors.ipynb
  - getting_started_with_SeqVec.ipynb
  - getting-started-with-SingleVariantProteinDataLoader.ipynb
- setup.py
- .pep8speaks.yml
- .dependabot
  - config.yml
- setup.cfg
- README.md
- tests
  - dont_test_4_integration.py
  - dataloaders
    - test_splicing.py
    - test_sequence.py
    - test_protein_dataloader.py
    - test_BedDataset.py
  - test_dataclasses.py
  - test_0_transforms_functional.py
  - benchmarking
    - benchmark.txt
  - extractors
    - test_vcf.py
    - test_vcf_query.py
    - test_vcf_matching.py
    - test_vcf_seq_extractor.py
    - test_protein.py
  - data
    - test_with_multiple_variants.vcf.gz.tbi
    - chr22_ENST00000319363_5UTR.vcf.gz
    - sample.gtf
    - test.vcf.gz
    - example_intervals.bed
    - dna_seq_enst_test2.txt
    - demo_proteins.pep.all.fa
    - singleSeq_vcf_enst_test2.vcf.gz
    - chr22_ENST00000319363_3UTR.alt_seqs.txt
    - demo_dna_seq.fa
    - uniprot_ref.txt
    - chr22_ENST00000319363_5UTR.alt_seqs.txt
    - singleVar_vcf_enst_test2.vcf.gz.tbi
    - sample_1_protein.gtf
    - test.vcf.gz.tbi
    - sample.fasta
    - chr22_ENST00000319363_5UTR.vcf.gz.tbi
    - test_with_multiple_variants.vcf.gz
    - chr22_ENST00000319363_3UTR.ref_seq.txt
    - chr22_ENST00000319363_3UTR.vcf.gz.tbi
    - subseq_sample.fasta
    - sample.5kb.fa
    - chr22_ENST00000319363.vcf.gz
    - sample_intervals_nochr.bed
    - sample_intervals.bed
    - vcf_file_for_testing_missense_mutations.vcf.gz.tbi
    - demo_dna_seq.fa.fai
    - singleVar_vcf_enst_test1_diff_type_of_variants.vcf.gz.tbi
    - vcf_file_for_testing_synonymous_mutations.vcf.gz
    - Output_singleSeq_vcf_enst_test2.txt
    - chr22_ENST00000319363_3UTR.vcf.gz
    - vcf_file_for_testing_synonymous_mutations.vcf.gz.tbi
    - Output_singleVar_vcf_enst_test2.txt
    - chr22_ENST00000319363_5UTR.ref_seq.txt
    - singleVar_vcf_enst_test2.vcf.gz
    - singleSeq_vcf_enst_test2.vcf.gz.tbi
    - singleVar_vcf_enst_test1_diff_type_of_variants.vcf.gz
    - vcf_file_for_testing_missense_mutations.vcf.gz
    - chr22_ENST00000319363.gtf
    - intervals_51bp.tsv
    - chr22_ENST00000319363.vcf.gz.tbi
    - dna_seq_enst_test1.txt
  - test_3_utils.py
  - test_translation_of_hg38
    - translation_confusuion_proteins.txt
    - test_translation.py
    - unresolved_proteins.txt
  - test_0_transforms_class.py
  - test_1_extractors.py
  - test_5_additional_tests_for_extractors.py
  - conftest.py
- requirements.txt
- .gitignore
- docs
  - Makefile
  - img
  - pydocmd.yml
  - kipoiseq
    - img
  - theme_dir
    - kipoiseq
      - img
    - css
      - extra.css
  - README.md
  - ipynb_pages.txt
  - render_ipynb.bash
- MANIFEST.in

kipoiseq

CircleCI

Coverage status

Standard set of data-loaders for training and making predictions for DNA sequence-based models.

All dataloaders in kipoiseq.dataloaders decorated with @kipoi_dataloader (SeqIntervalDl and StringSeqIntervalDl) are compatible Kipoi models and can be directly used when specifying a new model in model.yaml:

...
default_dataloader:
  defined_as: kipoiseq.dataloaders.SeqIntervalDl
  default_args:
    auto_resize_len: 1000 # override default args in SeqIntervalDl

dependencies:
  pip:
    - kipoiseq
...

Installation

pip install kipoiseq

Optional dependencies:

pip install cyvcf2, pyranges
conda install cyvcf2, pyranges

Getting started

from kipoiseq.dataloaders import SeqIntervalDl

dl = SeqIntervalDl.init_example()  # use the provided example files
# your own files
dl = SeqIntervalDl("intervals.bed", "genome.fa")

len(dl)  # length of the dataset

dl[0]  # get one instance. # returns a dictionary: 
# dict(inputs=<one-hot-encoded-array>, 
#      targets=<additional columns in the bed file>, 
#      metadata=dict(ranges=GenomicRanges(chr=, start, end)...

all = dl.load_all()  # load the whole dataset

# load batches of data
it = dl.batch_iter(32, num_workers=8)  # load batches of data in parallel using 8 workers
# returns a dictionary with all three keys: inputs, targets, metadata

it = dl.batch_train_iter(32, num_workers=8)
# returns a tuple: (inputs, targets), can be used directly with keras' `model.fit_generator`

More info:

Follow the getting-started colab notebook.
See docs

How to write your own data-loaders

Read the pytorch Data Loading and Processing Tutorial to become more familiar with transforms and dataloaders
Read the code for SeqIntervalDl in kipoiseq/dataloaders/sequence.py
- you can skip the @kipoi_dataloader and the long yaml doc-string. These are only required if you want to use dataloaders in Kipoi's model.yaml files.
Explore the available transforms (functional, class-based) or extractors (kipoiseq, genomelake)