Scala / Spark

How to Calculate total time taken for particular method in Spark[Code Snippet]

how-to-calculate -total-time-taken-for-method-in spark-24tutorials

In some cases where you applied Joins in the spark application, you might want to know the time taken to complete the particular join. Below code snippet might come in handy to achieve so.

import java.util.Date
val curent = new Date().getTime
println(curent)
Thread.sleep(30000)
val end = new Date().getTime
println(end)
println("time taken "+(end-curent).toFloat/60000 + "mins")

Output:
import java.util.Date
curent: Long = 1520502573995
end: Long = 1520502603996
time taken 0.5000167mins

All you need to do is get current time before method starts and get current time after method ends, then calculate the difference to get total time taken to complete that particular method.

Hope this code snippet helps!!

Share This Post

An Ambivert, music lover, enthusiast, artist, designer, coder, gamer, content writer. He is Professional Software Developer with hands-on experience in Spark, Kafka, Scala, Python, Hadoop, Hive, Sqoop, Pig, php, html,css. Know more about him at www.saikumar.me

Lost Password

Register