MarkDuplicatesSpark is not generating output file
REQUIRED for all errors and issues:
a) GATK version used: gatk-4.6.1.0
b) Exact command used: gatk MarkDuplicatesSpark -I SRR062641.paired.sam -O SRR062641_sorted_dedup.bam
c) Entire program log: Using GATK jar /home/meenameena048/softwares/gatk-4.6.1.0/gatk-package-4.6.1.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/meenameena048/softwares/gatk-4.6.1.0/gatk-package-4.6.1.0-local.jar MarkDuplicatesSpark -I SRR062641.paired.sam -O SRR062641_sorted_dedup.bam
14:15:25.730 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/meenameena048/softwares/gatk-4.6.1.0/gatk-package-4.6.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@7faa0680]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@4c635edc]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@7faa0680]
14:15:25.867 INFO MarkDuplicatesSpark - ------------------------------------------------------------
14:15:25.870 INFO MarkDuplicatesSpark - The Genome Analysis Toolkit (GATK) v4.6.1.0
14:15:25.870 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/
14:15:25.870 INFO MarkDuplicatesSpark - Executing as meenameena048@ubuntu1 on Linux v6.8.0-1017-gcp amd64
14:15:25.870 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v23.0.1+11-39
14:15:25.871 INFO MarkDuplicatesSpark - Start Date/Time: November 3, 2024, 2:15:25 PM UTC
14:15:25.871 INFO MarkDuplicatesSpark - ------------------------------------------------------------
14:15:25.871 INFO MarkDuplicatesSpark - ------------------------------------------------------------
14:15:25.872 INFO MarkDuplicatesSpark - HTSJDK Version: 4.1.3
14:15:25.872 INFO MarkDuplicatesSpark - Picard Version: 3.3.0
14:15:25.872 INFO MarkDuplicatesSpark - Built for Spark Version: 3.5.0
14:15:25.874 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2
14:15:25.875 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:15:25.875 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:15:25.875 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:15:25.875 INFO MarkDuplicatesSpark - Deflater: IntelDeflater
14:15:25.875 INFO MarkDuplicatesSpark - Inflater: IntelInflater
14:15:25.875 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20
14:15:25.876 INFO MarkDuplicatesSpark - Requester pays: disabled
14:15:25.876 INFO MarkDuplicatesSpark - Initializing engine
14:15:25.876 INFO MarkDuplicatesSpark - Done initializing engine
14:15:26.201 INFO SparkContext - Running Spark version 3.5.0
14:15:26.201 INFO SparkContext - OS info Linux, 6.8.0-1017-gcp, amd64
14:15:26.202 INFO SparkContext - Java version 23.0.1
14:15:26.348 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14:15:26.365 INFO MarkDuplicatesSpark - Shutting down engine
[November 3, 2024, 2:15:26 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=96468992
java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:334)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.createSparkContext(SparkContextFactory.java:185)
at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.getSparkContext(SparkContextFactory.java:117)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:28)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:150)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:203)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:222)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:166)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:209)
at org.broadinstitute.hellbender.Main.main(Main.java:306)
-
Hi charan
Our Spark tools are usually bound to the java version that is within our docker image which is 17 right now. It is possible that recent advances in java VM architecture is causing this error. Can you try running it with Java version 17?
Regards.
-
Thank you that worked out for me, java jdk 17 version has generated the output...
Please sign in to leave a comment.
2 comments