How to solve the issue of full disk utilization in HDFS Namenode

We hve ntied tht the disk ws 100% utilized in Nmende nd nt llwing ny further writes nd ls resulted in vrius errrs nd innsistenies.

We hve ntied tht the disk ws 100% utilized in Nmende nd nt llwing ny further writes nd ls resulted in vrius errrs nd innsistenies.
ur tem ws wndering why the Nmende disk ws full given tht it is nt sused t stre tul dt in its disk. Then we fund tht Nmende kees lgging ll the ertins(new blk, relitin, deletin et) in the EditLgs file whih tures the delt frm the lst FsImge file. FsImge ntins the mlete diretry struture (nmese) f the HDFS with detils but the ltin f the dt n the Dt Blks nd whih blks re stred n whih nde.
Nmende will kee n ending the ertins in EditLgs file(s) nd thse files will be mintined in disk until thse re tured s rt f the FsImge file. Nmende will rete the FsImge frm EditLgs nly during the strtu, but fter tht it wnt rete new FsImge file.
HDFS hs serte resses like hekintNde r SendryNmende resnsible fr reting new FsImge files(hekints) eridilly bsed n EditLgs nd lering ut thse EditLgs files.
When we run HDFS with Single mster in WS EMR, it desnt hve Sendry Nmende r hekint nde whih uses the Nmende t hve lt f EditLgs files in its Disk nd resulting full disk se utilistin.

rt frm disk utilistin, hving lt f EditLgs file will mke the Nmende restrt time t be lnger s the Nmende hs t rete new FsImge by running ver ll the trnstins tured in EditLgs.


Hw n we verme this?

When the HDFS is nfigured in H mde, the ress f eridi hekinting bsed n fs.hekint.erid nd fs.hekint.size nfigurtin by the stndby nde.
In the setu where weve single mster(similr t urs), weve ule f tins:
If the HDFS dwntime r mintenne windw is etble, we n trigger sveNmese in Nmende using fllwing mmnds.
hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode exit
In the ses where dwntime is nt etble, we need t strt hekintNde in nther mhine with sme HDFS nfigurtins. The hekint Nde will ensure reting these hekints utmtilly bsed n the erid nd size nfigurtins.

 Even when the hekint is mleted nd new FsImge is reted, the lder EditLg files my still be resent in the disk. It is ntrlled by nfigurtins dfs.namenode.num.extra.edits.retained and dfs.namenode.max.extra.edits.segments.retained. Ensure tht the vlues fr these nfigurtins re set t sensible numbers t vid unneessry disk usge in Nmende.

Share the knowledge

Still have questions?
Connect with us
Thank you.
Your request has been received.
Thank you!
The form has been submitted successfully.