Portable Hadoop Reader makes direct data flow possible - Xinhua | English.news.cn

CHICAGO, June 18 (Xinhua) -- Scientists working in Xian-He Sun's group at the Illinois Institute of Technology (IIT) have recently developed a cross-platform Hadoop reader known as PortHadoop, which has made data flow between Hadoop Distributed File System (HDFS) and Parallel File System (PFS) possible.

"PortHadoop, the system we developed, moves the data directly from the parallel file system to Hadoop's memory instead of copying from disk to disk," said Xian-He Sun, distinguished professor of computer science at the IIT.

It was difficult to move date between HDFS and PFS, and scientist who want to make the best use of analytics on Hadoop had to copy the data from PFS.

According to Sun, PortHadoop uses the concept of virtual blocks which map the data from PFS directly into Hadoop memory by creating a virtual HDFS environment.

These "virtual blocks" reside in the centralized namespace in HDFS NameNode. The HDFS MapReduce application cannot see the "virtual blocks"; a map task triggers the MPI file read procedure and fetches the data from the remote PFS before its Mapper function processes its data. In other words, a dexterous slight-of-hand from PortHadoop tricks the HDFS to skip the costly input/output operations and data replications it usually expects.

Sun sees PortHadoop as the consequence of the strong desire for scientists to merge high performance computing with cloud computing. With traditional scientific computing being merged with big data analytics, "it creates a bigger class of scientific computing that is badly needed to solve today's problems," Sun said.

Comments

Post a Comment

Popular posts from this blog

New Article Posted: Seven Things You Should Do in Your AMS

Given how advanced and thorough the modern association management software (AMS) system has become, there are many opportunities for your organization to leverage it. Here are seven ways I believe every association should be using their AMS. Click here to read the rest of the article . Of course, if you’re already on my  announcements list , you would have received notice of this new article right in your email box. Not signed up yet?  Click here to sign up . Go to the article
Read more

10 Common Mistakes Biblical Counselors Sometimes Make, Part 8

Image
A Word from Bob: You’re reading Part 8 of a 10-part blog series on 10 Common Mistakes Biblical Counselors Sometimes Make . Mistake #1: We Elevate Data Collection Above Soul Connection . Mistake #2: We Share God’s Eternal Story Before We Listen Well and Wisely to Our Friend’s Earthly Story . Mistake #3: We Talk at Counselees Rather Than Exploring Scriptures with Counselees . Mistake #4: We Practice Half-Biblical Counseling If We Address Sin but Neglect Suffering . Mistake #5: We Fail to Follow the Trinity’s Model of Comforting Care . Mistake #6: We Tend to View People One-Dimensionally . Mistake #7: We Devalue Emotions Rather Than Seeing Emotions as God’s Idea . Mistake #8: We Minimize the Complexity of the Soul-Body Interconnection Because we are soul physicians and soul care-givers, our tendency is to focus on matters of the soul. Most biblical counselors are not trained in the medical field, so we wisely avoid offering medical guidance. Yet, that appropr...
Read more