Error while running CreateReadCountPanelOfNormals
Hello!
When I run CreateReadCountPanelOfNormals I have the following error. I've created bins for the intervals with PreprocessIntervals and I've annotated them with AnnotateIntervals. Also, I used the counts of each sample from CollectReadCounts. However, when I run the same command with the same samples and intervals, without creating bins for them, everything works. Does anyone know why this happens or what I have to do to fix this error?
Thanks in advance!
(gatk) kate@kate-VirtualBox:~/gatk/gatk1$ ./gatk --java-options "-Xmx2400M" CreateReadCountPanelOfNormals \-I sample1b.counts.hdf5 \-I sample2b.counts.hdf5 \-I sample3b.counts.hdf5 \-I sample4b.counts.hdf5 \-I sample5b.counts.hdf5 \-I sample6b.counts.hdf5 \-I sample7b.counts.hdf5 \-I sample8b.counts.hdf5 \-I sample9b.counts.hdf5 \-I sample10b.counts.hdf5 \-I sample11b.counts.hdf5 \--annotated-intervals annotated_intervals_bins.tsv \--number-of-eigensamples 8 \--minimum-interval-median-percentile 20.0 \-O cnv.pon.bins.hdf5
python: Relink `/home/kate/miniconda3/envs/gatk/bin/../lib/libgfortran.so.4' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Using GATK jar /home/kate/gatk/gatk1/gatk-package-4.1.4.1-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx2400M -jar /home/kate/gatk/gatk1/gatk-package-4.1.4.1-local.jar CreateReadCountPanelOfNormals -I sample1b.counts.hdf5 -I sample2b.counts.hdf5 -I sample3b.counts.hdf5 -I sample4b.counts.hdf5 -I sample5b.counts.hdf5 -I sample6b.counts.hdf5 -I sample7b.counts.hdf5 -I sample8b.counts.hdf5 -I sample9b.counts.hdf5 -I sample10b.counts.hdf5 -I sample11b.counts.hdf5 --annotated-intervals annotated_intervals_bins.tsv --number-of-eigensamples 8 --minimum-interval-median-percentile 20.0 -O cnv.pon.bins.hdf5
16:46:22.901 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/kate/gatk/gatk1/gatk-package-4.1.4.1-local.jar!/com/intel/gkl/native/libgkl_compression.so
Feb 25, 2020 4:46:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
16:46:25.496 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
16:46:25.504 INFO CreateReadCountPanelOfNormals - The Genome Analysis Toolkit (GATK) v4.1.4.1
16:46:25.517 INFO CreateReadCountPanelOfNormals - For support and documentation go to https://software.broadinstitute.org/gatk/
16:46:25.524 INFO CreateReadCountPanelOfNormals - Executing as kate@kate-VirtualBox on Linux v4.15.0-88-generic amd64
16:46:25.525 INFO CreateReadCountPanelOfNormals - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
16:46:25.525 INFO CreateReadCountPanelOfNormals - Start Date/Time: February 25, 2020 4:46:22 PM EET
16:46:25.526 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
16:46:25.527 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
16:46:25.537 INFO CreateReadCountPanelOfNormals - HTSJDK Version: 2.21.0
16:46:25.545 INFO CreateReadCountPanelOfNormals - Picard Version: 2.21.2
16:46:25.549 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.COMPRESSION_LEVEL : 2
16:46:25.564 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
16:46:25.565 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
16:46:25.565 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
16:46:25.565 INFO CreateReadCountPanelOfNormals - Deflater: IntelDeflater
16:46:25.566 INFO CreateReadCountPanelOfNormals - Inflater: IntelInflater
16:46:25.566 INFO CreateReadCountPanelOfNormals - GCS max retries/reopens: 20
16:46:25.573 INFO CreateReadCountPanelOfNormals - Requester pays: disabled
16:46:25.576 INFO CreateReadCountPanelOfNormals - Initializing engine
16:46:25.576 INFO CreateReadCountPanelOfNormals - Done initializing engine
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/02/25 16:46:29 WARN Utils: Your hostname, kate-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
20/02/25 16:46:29 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/02/25 16:46:30 INFO SparkContext: Running Spark version 2.4.3
16:46:32.194 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/25 16:46:32 INFO SparkContext: Submitted application: CreateReadCountPanelOfNormals
20/02/25 16:46:33 INFO SecurityManager: Changing view acls to: kate
20/02/25 16:46:33 INFO SecurityManager: Changing modify acls to: kate
20/02/25 16:46:33 INFO SecurityManager: Changing view acls groups to:
20/02/25 16:46:33 INFO SecurityManager: Changing modify acls groups to:
20/02/25 16:46:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kate); groups with view permissions: Set(); users with modify permissions: Set(kate); groups with modify permissions: Set()
20/02/25 16:46:35 INFO Utils: Successfully started service 'sparkDriver' on port 33013.
20/02/25 16:46:35 INFO SparkEnv: Registering MapOutputTracker
20/02/25 16:46:35 INFO SparkEnv: Registering BlockManagerMaster
20/02/25 16:46:35 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/02/25 16:46:35 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/02/25 16:46:36 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-cd2e6e35-4365-4826-91ed-8ed487781b2e
20/02/25 16:46:36 INFO MemoryStore: MemoryStore started with capacity 1212.0 MB
20/02/25 16:46:36 INFO SparkEnv: Registering OutputCommitCoordinator
20/02/25 16:46:37 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/02/25 16:46:38 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.2.15:4040
20/02/25 16:46:39 INFO Executor: Starting executor ID driver on host localhost
20/02/25 16:46:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34563.
20/02/25 16:46:39 INFO NettyBlockTransferService: Server created on 10.0.2.15:34563
20/02/25 16:46:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/02/25 16:46:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:46:39 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.2.15:34563 with 1212.0 MB RAM, BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:46:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:46:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.2.15, 34563, None)
16:46:40.810 INFO CreateReadCountPanelOfNormals - Spark verbosity set to INFO (see --spark-verbosity argument)
20/02/25 16:46:40 INFO HDF5Library: Trying to load HDF5 library from:
jar:file:/home/kate/gatk/gatk1/gatk-package-4.1.4.1-local.jar!/org/broadinstitute/hdf5/libjhdf5.2.11.0.so
20/02/25 16:46:41 INFO H5: HDF5 library:
20/02/25 16:46:41 INFO H5: successfully loaded.
16:46:41.262 INFO CreateReadCountPanelOfNormals - Retrieving intervals from first read-counts file (sample1b.counts.hdf5)...
16:46:51.392 INFO CreateReadCountPanelOfNormals - Reading and validating annotated intervals...
16:48:00.526 INFO CreateReadCountPanelOfNormals - Validating and aggregating input read-counts files...
16:48:00.656 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample1b.counts.hdf5 (1 / 11)
16:48:24.266 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample2b.counts.hdf5 (2 / 11)
16:48:31.296 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample3b.counts.hdf5 (3 / 11)
16:48:35.259 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample4b.counts.hdf5 (4 / 11)
16:50:03.437 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample5b.counts.hdf5 (5 / 11)
16:50:11.426 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample6b.counts.hdf5 (6 / 11)
16:50:17.338 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample7b.counts.hdf5 (7 / 11)
16:50:24.532 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample8b.counts.hdf5 (8 / 11)
16:50:31.728 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample9b.counts.hdf5 (9 / 11)
16:52:02.499 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample10b.counts.hdf5 (10 / 11)
16:52:28.243 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file sample11b.counts.hdf5 (11 / 11)
16:52:39.990 INFO CreateReadCountPanelOfNormals - Creating the panel of normals...
16:52:40.243 INFO HDF5SVDReadCountPanelOfNormals - Creating read-count panel of normals at /home/kate/gatk/gatk1/cnv.pon.bins.hdf5...
16:52:41.148 INFO HDF5SVDReadCountPanelOfNormals - Writing version number (7.0)...
16:52:41.448 INFO HDF5SVDReadCountPanelOfNormals - Writing command line...
16:52:41.695 INFO HDF5SVDReadCountPanelOfNormals - Writing sequence dictionary...
16:52:41.755 INFO HDF5SVDReadCountPanelOfNormals - Writing original read counts (2897706 x 11)...
20/02/25 16:55:10 WARN HeartbeatReceiver: Removing executor driver with no recent heartbeats: 143286 ms exceeds timeout 120000 ms
20/02/25 16:55:11 ERROR TaskSchedulerImpl: Lost an executor driver (already removed): Executor heartbeat timed out after 143286 ms
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 WARN SparkContext: Killing executors is not supported by current scheduler.
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:13 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:13 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:14 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:14 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:14 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:14 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:14 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:14 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:17 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:17 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:17 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:27 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:27 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:27 INFO BlockManager: Reporting 0 blocks to the master.
16:55:28.468 INFO HDF5SVDReadCountPanelOfNormals - Writing original sample filenames (11)...
16:55:28.495 INFO HDF5SVDReadCountPanelOfNormals - Writing original intervals (2897706)...
16:55:34.706 INFO HDF5SVDReadCountPanelOfNormals - Writing GC-content annotations for original intervals (2897706)...
16:55:35.816 INFO HDF5SVDReadCountPanelOfNormals - Preprocessing and standardizing read counts...
16:55:35.905 INFO SVDDenoisingUtils - Preprocessing read counts...
16:55:35.905 INFO SVDDenoisingUtils - Transforming read counts to fractional coverage...
20/02/25 16:55:37 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:37 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:37 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:47 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:47 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:47 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:55:57 INFO Executor: Told to re-register on heartbeat
20/02/25 16:55:57 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:55:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:55:57 INFO BlockManager: Reporting 0 blocks to the master.
16:55:57.514 INFO SVDDenoisingUtils - Performing GC-bias correction...
20/02/25 16:56:07 INFO Executor: Told to re-register on heartbeat
20/02/25 16:56:07 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:56:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:56:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:56:07 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:51 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:51 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:51 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:52 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:52 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:52 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:52 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:52 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:52 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:52 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:52 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:52 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:57:57 INFO Executor: Told to re-register on heartbeat
20/02/25 16:57:57 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:57:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:57:57 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:07 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:07 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:07 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:17 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:17 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:17 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:27 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:27 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:27 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:38 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:38 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:38 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:54 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:54 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:54 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:54 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:54 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:58:57 INFO Executor: Told to re-register on heartbeat
20/02/25 16:58:57 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:58:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:58:57 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:59:07 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:07 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:07 INFO BlockManager: Reporting 0 blocks to the master.
16:59:13.858 INFO SVDDenoisingUtils - Filtering intervals with median (across samples) less than or equal to the 20.00 percentile...
16:59:15.948 INFO SVDDenoisingUtils - After filtering, 2897706 out of 2897706 intervals remain...
16:59:15.949 INFO SVDDenoisingUtils - Dividing by interval medians...
20/02/25 16:59:17 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:17 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:17 INFO BlockManager: Reporting 0 blocks to the master.
16:59:23.511 INFO SVDDenoisingUtils - Filtering samples with a fraction of zero-coverage intervals above 5.00 percent...
16:59:26.261 INFO SVDDenoisingUtils - After filtering, 11 out of 11 samples remain...
16:59:26.261 INFO SVDDenoisingUtils - Filtering intervals with a fraction of zero-coverage samples above 5.00 percent...
20/02/25 16:59:27 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:27 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:27 INFO BlockManager: Reporting 0 blocks to the master.
16:59:30.934 INFO SVDDenoisingUtils - After filtering, 2897706 out of 2897706 intervals remain...
16:59:30.936 INFO SVDDenoisingUtils - Filtering samples with a median (across intervals) below the 2.50 percentile or above the 97.50 percentile...
20/02/25 16:59:37 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:37 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:37 INFO BlockManager: Reporting 0 blocks to the master.
16:59:42.914 INFO SVDDenoisingUtils - After filtering, 11 out of 11 samples remain...
20/02/25 16:59:54 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:54 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:54 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:54 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:54 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 16:59:57 INFO Executor: Told to re-register on heartbeat
20/02/25 16:59:57 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 16:59:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 16:59:57 INFO BlockManager: Reporting 0 blocks to the master.
16:59:58.291 INFO SVDDenoisingUtils - Heap utilization statistics [MB]:
16:59:58.316 INFO SVDDenoisingUtils - Used memory: 1557
16:59:58.317 INFO SVDDenoisingUtils - Free memory: 762
16:59:58.320 INFO SVDDenoisingUtils - Total memory: 2320
16:59:58.321 INFO SVDDenoisingUtils - Maximum memory: 2320
16:59:58.322 INFO SVDDenoisingUtils - Performing garbage collection...
17:01:33.244 INFO SVDDenoisingUtils - Heap utilization statistics [MB]:
17:01:33.249 INFO SVDDenoisingUtils - Used memory: 1400
17:01:33.249 INFO SVDDenoisingUtils - Free memory: 919
17:01:33.249 INFO SVDDenoisingUtils - Total memory: 2320
17:01:33.249 INFO SVDDenoisingUtils - Maximum memory: 2320
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:33 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:33 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:33 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:33 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:37 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:37 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:37 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:47 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:47 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:47 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:01:57 INFO Executor: Told to re-register on heartbeat
20/02/25 17:01:57 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:01:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:01:57 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:02:07 INFO Executor: Told to re-register on heartbeat
20/02/25 17:02:07 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:02:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:07 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:02:18 INFO Executor: Told to re-register on heartbeat
20/02/25 17:02:18 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:02:18 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:18 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:18 INFO BlockManager: Reporting 0 blocks to the master.
17:02:26.870 INFO SVDDenoisingUtils - 0 zero-coverage values were imputed to the median of the non-zero values in the corresponding interval...
20/02/25 17:02:27 INFO Executor: Told to re-register on heartbeat
20/02/25 17:02:27 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:02:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:27 INFO BlockManager: Reporting 0 blocks to the master.
20/02/25 17:02:41 INFO Executor: Told to re-register on heartbeat
20/02/25 17:02:41 INFO BlockManager: BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None) re-registering with master
20/02/25 17:02:41 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:41 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.2.15, 34563, None)
20/02/25 17:02:41 INFO BlockManager: Reporting 0 blocks to the master.
17:02:44.690 INFO SVDDenoisingUtils - 0 values below the 0.10 percentile or above the 99.90 percentile were truncated to the corresponding value...
17:02:44.700 INFO SVDDenoisingUtils - Panel read counts preprocessed.
17:02:44.700 INFO SVDDenoisingUtils - Standardizing read counts...
17:02:44.701 INFO SVDDenoisingUtils - Dividing by sample medians and transforming to log2 space...
17:02:45.773 WARN HDF5SVDReadCountPanelOfNormals - Exception encountered during creation of panel of normals (java.lang.IllegalArgumentException: Sample at index 0 does not have a non-negative sample median.). Attempting to delete partial output in /home/kate/gatk/gatk1/cnv.pon.bins.hdf5...
20/02/25 17:02:46 INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
20/02/25 17:02:46 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/02/25 17:02:46 INFO MemoryStore: MemoryStore cleared
20/02/25 17:02:46 INFO BlockManager: BlockManager stopped
20/02/25 17:02:46 INFO BlockManagerMaster: BlockManagerMaster stopped
20/02/25 17:02:46 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/02/25 17:02:46 INFO SparkContext: Successfully stopped SparkContext
17:02:46.961 INFO CreateReadCountPanelOfNormals - Shutting down engine
[February 25, 2020 5:02:46 PM EET] org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals done. Elapsed time: 16.41 minutes.
Runtime.totalMemory()=2432696320
org.broadinstitute.hellbender.exceptions.GATKException: Could not create panel of normals. It may be necessary to use stricter parameters for filtering. For example, use a larger value of minimum-interval-median-percentile.
at org.broadinstitute.hellbender.tools.copynumber.denoising.HDF5SVDReadCountPanelOfNormals.create(HDF5SVDReadCountPanelOfNormals.java:354)
at org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals.runPipeline(CreateReadCountPanelOfNormals.java:290)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
at org.broadinstitute.hellbender.Main.main(Main.java:292)
Caused by: java.lang.IllegalArgumentException: Sample at index 0 does not have a non-negative sample median.
at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:725)
at org.broadinstitute.hellbender.utils.param.ParamUtils.isPositive(ParamUtils.java:165)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.lambda$divideBySampleMedianAndTransformToLog2$27(SVDDenoisingUtils.java:483)
at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.divideBySampleMedianAndTransformToLog2(SVDDenoisingUtils.java:482)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.preprocessAndStandardizePanel(SVDDenoisingUtils.java:86)
at org.broadinstitute.hellbender.tools.copynumber.denoising.HDF5SVDReadCountPanelOfNormals.create(HDF5SVDReadCountPanelOfNormals.java:288)
... 8 more
20/02/25 17:02:47 INFO ShutdownHookManager: Shutdown hook called
20/02/25 17:02:47 INFO ShutdownHookManager: Deleting directory /tmp/spark-89a4d583-c15b-4b57-abf2-6f41aec727c7
-
Hi Kate Pal,
I was a little confused by your post. I am assuming the problem is with CreateReadCountPanelOfNormals tool, not DetermineGermlineContigPloidy?
If so, it looks like the exception that causes the tool to fail is this:
"17:02:45.773 WARN HDF5SVDReadCountPanelOfNormals - Exception encountered during creation of panel of normals (java.lang.IllegalArgumentException: Sample at index 0 does not have a non-negative sample median.). Attempting to delete partial output in /home/kate/gatk/gatk1/cnv.pon.bins.hdf5..."
I recommend looking at your count files (you can use hdf5 viewer for that). Are most of the counts 0? Do you know if the samples you are analyzing are exomes? If so, then you need to provide PreprocessIntervals with a list of targets for the capture kit of the exomes. Otherwise, PreprocessIntervals generates a bin list that spans entire genome, so most of your bins will have count 0.
Hope that helps!
-
Sorry, CreateReadCountPanelOfNormals I wanted to write. I edited the post now.
Maybe this is the problem. I don't know if the samples are exomes. Anyway, I'll install hdf5 viewer to see the counts and then try to run the command again. Hope that works!
Thanks for your quick response!
-
Hi Andrey Smirnov and Kate Pal!
Thanks for describing the issue and for the answer!
I have a similar error (data = panel sequencing, T/N).
I inspected counts dataset in the hdf5 object. 9138 counts of 25k are non-zero, and the median is 0,
so it is indeed the reason of the exception in my case.
hdf5 object was created by:
gatk PreprocessIntervals \
-R hg38.fa \
--interval-merging-rule OVERLAPPING_ONLY \
-O target.interval_list \
-L coverage.bed \
--bin-length 0 \
--padding 250
gatk CollectReadCounts \
-I normal.bam \
-L target.interval_list \
--interval-merging-rule OVERLAPPING_ONLY \
-O target-coverage.hdf5 --format HDF5Could you suggest how may I alter these two steps to avoid the exception?
Sergey
-
`CreateReadCountPanelOfNormals` has multiple arguments for filtering input count matrix, such as `--minimum-interval-median-percentile`, `--maximum-zeros-in-sample-percentage` etc -- you can see a full list if you invoke the tool with the `--help` argument.
You can use this to filter input counts, and get rid of zero median samples in your input.
P.S. I also just changed the error messaged, it should be "Sample at index %s does not have a positive sample median." -- it will be in the next release.
-
Andrey Smirnov, thanks for the answer!
In my use case, I'm calling CNVs in Tumor-Normal pairs (T/N), so I have just one normal to create a PON,
and I cannot filter out that one normal.
Many T/N pairs passed through the pipeline, and just one failed.
Interestingly, one of the successful samples had 9157/24745 = 37% intervals with non-zero counts in the hdf5 file, and the failed one had 9138/24745 = 36.9%, so the difference is not that big.
Is there a way to relax filters a bit to create PON in such a case?
I looked through the parameters, and the only one acting sample-wise rather than interval-wise is
--maximum-zeros-in-sample-percentage. But obviously it works to filter out samples, not to include samples excluded by the other filters. I tried setting it to 100 and got the same error.
Sergey
-
Hi Sergey Naumenko,
Thanks for posting your issues! A few things:
1) It may be helpful if you look at (or post) the full log. There will be messages concerning the filtering (which you can see in Kate Pal's post above) that may be helpful in pinpointing which filter to adjust. (I'm a little confused as to why setting maximum-zeros-in-sample-percentage to 100 doesn't resolve things, but perhaps the log will help clear things up. Hopefully, this is not a bug!)
2) The fact that only 9k of your 25k intervals are non-zero in multiple samples is a bit concerning. Are you sure you are using a suitable interval list for your targets? It may be good to check the distribution of this quantity across samples to make sure it looks reasonable.
3) It sounds like you are running multiple T/N pairs, but are building separate PoNs for each pair. Unless you expect different patterns of systematic bias/noise in each pair (e.g., they were all sequenced using different protocols), this seems like an idiosyncratic use of the pipeline. Typically, we'd expect that the normals for all pairs sequenced using the same protocol can be used to build a single PoN.
-
Hi Samuel Lee!
Thanks for the answer!
1) I am posting the log below. In the light of 2) and 3) there is not much sense to chase it, unless you want --maximum-zeros-in-sample-percentage 100 to work in a permissive mode.
2-3) A very good point, thanks! I think it is because we coupled CNV calling with T/N small variant calling, hence the extended intervals (in the exome or in the panels you always get some extra coverage outside of the regions and it is possible to pick up some more variants) and T/N CNV pairing. Obviously we should decouple CNV from small variants it and rethink our workflow.
Sergey
ps. Log:
15:45:23.220 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
May 22, 2020 3:45:23 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
15:45:23.354 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
15:45:23.354 INFO CreateReadCountPanelOfNormals - The Genome Analysis Toolkit (GATK) v4.1.7.0
15:45:23.354 INFO CreateReadCountPanelOfNormals - For support and documentation go to https://software.broadinstitute.org/gatk/
15:45:23.354 INFO CreateReadCountPanelOfNormals - Executing as ubuntu@ on Linux v5.3.0-1019-aws amd64
15:45:23.354 INFO CreateReadCountPanelOfNormals - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
15:45:23.354 INFO CreateReadCountPanelOfNormals - Start Date/Time: May 22, 2020 3:45:23 PM UTC
15:45:23.354 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
15:45:23.354 INFO CreateReadCountPanelOfNormals - ------------------------------------------------------------
15:45:23.355 INFO CreateReadCountPanelOfNormals - HTSJDK Version: 2.21.2
15:45:23.355 INFO CreateReadCountPanelOfNormals - Picard Version: 2.21.9
15:45:23.355 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.COMPRESSION_LEVEL : 2
15:45:23.355 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:45:23.355 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:45:23.355 INFO CreateReadCountPanelOfNormals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:45:23.355 INFO CreateReadCountPanelOfNormals - Deflater: IntelDeflater
15:45:23.355 INFO CreateReadCountPanelOfNormals - Inflater: IntelInflater
15:45:23.355 INFO CreateReadCountPanelOfNormals - GCS max retries/reopens: 20
15:45:23.355 INFO CreateReadCountPanelOfNormals - Requester pays: disabled
15:45:23.355 INFO CreateReadCountPanelOfNormals - Initializing engine
15:45:23.355 INFO CreateReadCountPanelOfNormals - Done initializing engine
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/05/22 15:45:23 INFO SparkContext: Running Spark version 2.4.3
15:45:23.650 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/05/22 15:45:23 INFO SparkContext: Submitted application: CreateReadCountPanelOfNormals
20/05/22 15:45:23 INFO SecurityManager: Changing view acls to: ubuntu
20/05/22 15:45:23 INFO SecurityManager: Changing modify acls to: ubuntu
20/05/22 15:45:23 INFO SecurityManager: Changing view acls groups to:
20/05/22 15:45:23 INFO SecurityManager: Changing modify acls groups to:
20/05/22 15:45:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); groups with view permissions: Set(); users with modify permissions: Set(ubuntu); groups with modify permissions: Set()
20/05/22 15:45:23 INFO Utils: Successfully started service 'sparkDriver' on port 34533.
20/05/22 15:45:23 INFO SparkEnv: Registering MapOutputTracker
20/05/22 15:45:23 INFO SparkEnv: Registering BlockManagerMaster
20/05/22 15:45:23 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/05/22 15:45:23 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/05/22 15:45:23 INFO DiskBlockManager: Created local directory at /data/2020-05-11_gatkcnv/gatkcnv_failed/blockmgr-cc70fdf2-fbd0-4987-be5a-e7d7e7e39918
20/05/22 15:45:23 INFO MemoryStore: MemoryStore started with capacity 74.5 GB
20/05/22 15:45:23 INFO SparkEnv: Registering OutputCommitCoordinator
20/05/22 15:45:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/05/22 15:45:24 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://
20/05/22 15:45:24 INFO Executor: Starting executor ID driver on host localhost
20/05/22 15:45:24 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34803.
20/05/22 15:45:24 INFO NettyBlockTransferService: Server created on
20/05/22 15:45:24 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/05/22 15:45:24 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver,
20/05/22 15:45:24 INFO BlockManagerMasterEndpoint: Registering block manager
20/05/22 15:45:24 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver,
20/05/22 15:45:24 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver,
15:45:24.258 INFO CreateReadCountPanelOfNormals - Spark verbosity set to INFO (see --spark-verbosity argument)
20/05/22 15:45:24 INFO HDF5Library: Trying to load HDF5 library from:
jar:file:/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar!/org/broadinstitute/hdf5/libjhdf5.2.11.0.so
20/05/22 15:45:24 INFO H5: HDF5 library:
20/05/22 15:45:24 INFO H5: successfully loaded.
15:45:24.301 WARN CreateReadCountPanelOfNormals - Number of eigensamples (20) is greater than the number of input samples (1); the number of samples retained after filtering will be used instead.
15:45:24.303 INFO CreateReadCountPanelOfNormals - Retrieving intervals from first read-counts file (S_N-sort.hdf5)...
15:45:24.453 INFO CreateReadCountPanelOfNormals - Reading and validating annotated intervals...
15:45:24.709 INFO CreateReadCountPanelOfNormals - Validating and aggregating input read-counts files...
15:45:24.710 INFO CreateReadCountPanelOfNormals - Aggregating read-counts file S_N-sort.hdf5 (1 / 1)
15:45:24.759 INFO CreateReadCountPanelOfNormals - Creating the panel of normals...
15:45:24.760 INFO HDF5SVDReadCountPanelOfNormals - Creating read-count panel of normals at /data/2020-05-11_gatkcnv/gatkcnv_failed/S_N-sort.pon.hdf5...
15:45:24.761 INFO HDF5SVDReadCountPanelOfNormals - Writing version number (7.0)...
15:45:24.766 INFO HDF5SVDReadCountPanelOfNormals - Writing command line...
15:45:24.771 INFO HDF5SVDReadCountPanelOfNormals - Writing sequence dictionary...
15:45:24.780 INFO HDF5SVDReadCountPanelOfNormals - Writing original read counts (24745 x 1)...
15:45:24.784 INFO HDF5SVDReadCountPanelOfNormals - Writing original sample filenames (1)...
15:45:24.784 INFO HDF5SVDReadCountPanelOfNormals - Writing original intervals (24745)...
15:45:24.797 INFO HDF5SVDReadCountPanelOfNormals - Writing GC-content annotations for original intervals (24745)...
15:45:24.797 INFO HDF5SVDReadCountPanelOfNormals - Preprocessing and standardizing read counts...
15:45:24.798 INFO SVDDenoisingUtils - Preprocessing read counts...
15:45:24.798 INFO SVDDenoisingUtils - Transforming read counts to fractional coverage...
15:45:24.802 INFO SVDDenoisingUtils - Performing GC-bias correction...
15:45:24.835 INFO SVDDenoisingUtils - Filtering intervals with median (across samples) less than or equal to the 10.00 percentile...
15:45:24.844 INFO SVDDenoisingUtils - After filtering, 24745 out of 24745 intervals remain...
15:45:24.844 INFO SVDDenoisingUtils - Dividing by interval medians...
15:45:24.860 INFO SVDDenoisingUtils - A value of 100 was provided for argument maximum-zeros-in-sample-percentage, so the corresponding filtering step will be skipped...
15:45:24.860 INFO SVDDenoisingUtils - Filtering intervals with a fraction of zero-coverage samples above 5.00 percent...
15:45:24.877 INFO SVDDenoisingUtils - After filtering, 24745 out of 24745 intervals remain...
15:45:24.877 INFO SVDDenoisingUtils - Filtering samples with a median (across intervals) below the 2.50 percentile or above the 97.50 percentile...
15:45:24.885 INFO SVDDenoisingUtils - After filtering, 1 out of 1 samples remain...
15:45:24.894 INFO SVDDenoisingUtils - Heap utilization statistics [MB]:
15:45:24.894 INFO SVDDenoisingUtils - Used memory: 156
15:45:24.894 INFO SVDDenoisingUtils - Free memory: 327
15:45:24.895 INFO SVDDenoisingUtils - Total memory: 483
15:45:24.895 INFO SVDDenoisingUtils - Maximum memory: 127464
15:45:24.895 INFO SVDDenoisingUtils - Performing garbage collection...
15:45:24.975 INFO SVDDenoisingUtils - Heap utilization statistics [MB]:
15:45:24.975 INFO SVDDenoisingUtils - Used memory: 53
15:45:24.975 INFO SVDDenoisingUtils - Free memory: 429
15:45:24.975 INFO SVDDenoisingUtils - Total memory: 483
15:45:24.975 INFO SVDDenoisingUtils - Maximum memory: 127464
15:45:24.995 INFO SVDDenoisingUtils - 0 zero-coverage values were imputed to the median of the non-zero values in the corresponding interval...
15:45:24.998 INFO SVDDenoisingUtils - 0 values below the 0.10 percentile or above the 99.90 percentile were truncated to the corresponding value...
15:45:24.998 INFO SVDDenoisingUtils - Panel read counts preprocessed.
15:45:24.998 INFO SVDDenoisingUtils - Standardizing read counts...
15:45:24.998 INFO SVDDenoisingUtils - Dividing by sample medians and transforming to log2 space...
15:45:25.000 WARN HDF5SVDReadCountPanelOfNormals - Exception encountered during creation of panel of normals (java.lang.IllegalArgumentException: Sample does not have a non-negative sample median.). Attempting to delete partial output in /data/2020-05-11_gatkcnv/gatkcnv_failed/S_N-sort.pon.hdf5...
20/05/22 15:45:25 INFO SparkUI: Stopped Spark web UI at http://
20/05/22 15:45:25 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/05/22 15:45:25 INFO MemoryStore: MemoryStore cleared
20/05/22 15:45:25 INFO BlockManager: BlockManager stopped
20/05/22 15:45:25 INFO BlockManagerMaster: BlockManagerMaster stopped
20/05/22 15:45:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/05/22 15:45:25 INFO SparkContext: Successfully stopped SparkContext
15:45:25.030 INFO CreateReadCountPanelOfNormals - Shutting down engine
[May 22, 2020 3:45:25 PM UTC] org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals done. Elapsed time: 0.03 minutes.
Runtime.totalMemory()=506986496
org.broadinstitute.hellbender.exceptions.GATKException: Could not create panel of normals. It may be necessary to use stricter parameters for filtering. For example, use a larger value of minimum-interval-median-percentile.
at org.broadinstitute.hellbender.tools.copynumber.denoising.HDF5SVDReadCountPanelOfNormals.create(HDF5SVDReadCountPanelOfNormals.java:354)
at org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals.runPipeline(CreateReadCountPanelOfNormals.java:290)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
at org.broadinstitute.hellbender.Main.main(Main.java:292)
Caused by: java.lang.IllegalArgumentException: Sample does not have a non-negative sample median.
at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:727)
at org.broadinstitute.hellbender.utils.param.ParamUtils.isPositive(ParamUtils.java:165)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.lambda$divideBySampleMedianAndTransformToLog2$27(SVDDenoisingUtils.java:483)
at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:557)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.divideBySampleMedianAndTransformToLog2(SVDDenoisingUtils.java:482)
at org.broadinstitute.hellbender.tools.copynumber.denoising.SVDDenoisingUtils.preprocessAndStandardizePanel(SVDDenoisingUtils.java:86)
at org.broadinstitute.hellbender.tools.copynumber.denoising.HDF5SVDReadCountPanelOfNormals.create(HDF5SVDReadCountPanelOfNormals.java:288)
... 8 more
20/05/22 15:45:25 INFO ShutdownHookManager: Shutdown hook called
20/05/22 15:45:25 INFO ShutdownHookManager: Deleting directory /data/2020-05-11_gatkcnv/gatkcnv_failed/spark-fa02e218-d5a7-4879-88c9-e433b3fb1b52
Using GATK jar /share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms500m -Xmx131859m -XX:+UseSerialGC -Djava.io.tmpdir=. -jar /share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar CreateReadCountPanelOfNormals -O S_N-sort.pon.hdf5 --annotated-intervals coverage.gcannotated.tsv -I S_N-sort.hdf5 --maximum-zeros-in-sample-percentage 100 -
Hi Samuel Lee!
I removed padding in the PreprocessIntervals step:
gatk PreprocessIntervals \
-R hg38.fa \
--interval-merging-rule OVERLAPPING_ONLY \
-O target.interval_list \
-L coverage.bed \
--bin-length 0 \
--padding 0This decreased the amount of non-zero elements in the hdf5 obect to 8259/24745 = 33%,
but this time CreateReadCountPanelofNormals passed, so no zero median exception.
Could you please expand more on the padding logic? What is the motivation of 250bp + 250bp padding?
Thanks!
Sergey
-
Hi Sergey Naumenko,
Thanks for sharing the log. I now see that there is a slight bug in the rounding logic for filtering intervals. I'd expect this filter to drop the zero-coverage intervals:
15:45:24.860 INFO SVDDenoisingUtils - Filtering intervals with a fraction of zero-coverage samples above 5.00 percent...
since 100% of samples have zero coverage in these intervals (in this case, one sample). Let me issue a PR to try to fix this.
As for padding, the idea is simply to pick up the tails of the coverage distribution around each target interval. When padding causes adjacent target intervals to overlap, there is some logic performed to split the padding evenly between the intervals, which may have led to the slight change in your sample median.
In any case, I'd rethink whether it is appropriate to use single samples to build PoNs for each pair, and double check that the target list is suitable---that seems like a lot of zero-coverage intervals.
-
Thanks again, Samuel Lee!
I successfully ran CNV detection for 10 T/N pairs along the lines of the tutorial
Briefly, the steps are:
- PreprocessIntervals
- AnnotateIntervals (gc content)
- CollectReadCounts (for every sample)
- CreateReadCountPanelOfNormals = pon.hdf5
- DenoiseReadCounts (for every sample vs pon.hdf5)
- CollectAllelicCounts (for every sample)
- ModelSegments (for every tumor sample with T and N allelic counts)
- CallCopyRatioSegments (for every tumor sample)
- PlotModeledSegments
It all went well, and now I'm trying to modify workflow for tumor only calling.
7. ModelSegments (for every tumor sample)
gatk --java-options "-Xmx4g" ModelSegments \
--allelic-counts S1_T.allelicCounts.tsv \
--output . \
--output-prefix S1_T8. CallCopyRatioSegments
9. PlotModeledSegments.
On the last step I've got the error:
java.lang.IllegalArgumentException: Number of denoised-copy-ratio points in input modeled-segments file is inconsistent with that in input denoised-copy-ratios file.
at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:727)
at org.broadinstitute.hellbender.tools.copynumber.plotting.PlotModeledSegments.validateNumPointsPerContig(PlotModeledSegments.java:273)
at org.broadinstitute.hellbender.tools.copynumber.plotting.PlotModeledSegments.doWork(PlotModeledSegments.java:209)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
at org.broadinstitute.hellbender.Main.main(Main.java:292)I tried both syntaxis options from:
https://gatk.broadinstitute.org/hc/en-us/articles/360041416172-PlotModeledSegments
gatk PlotModeledSegments \ --denoised-copy-ratios tumor.denoisedCR.tsv \ --allelic-counts tumor.hets.tsv \ --segments tumor.modelFinal.seg \ --sequence-dictionary contigs_to_plot.dict \ --output-prefix tumor \ -O output_dirand
gatk PlotModeledSegments \ --denoised-copy-ratios tumor.denoisedCR.tsv \ --segments tumor.modelFinal.seg \ --sequence-dictionary contigs_to_plot.dict \ --output-prefix tumor \ -O output_dirCould you please explain, how can I tune-up SegmentPlotting for tumor only case
(with the panel of normals but without matched T/N).
Thanks!
Sergey
-
Hi Sergey Naumenko,
Looks like you may have neglected to include your copy ratios in your ModelSegments call; try this:
gatk --java-options "-Xmx4g" ModelSegments \
--denoised-copy-ratios tumor.denoisedCR.tsv \
--allelic-counts S1_T.allelicCounts.tsv \
--output . \
--output-prefix S1_T
Please sign in to leave a comment.
11 comments