非常快的文本文件处理(C)

我写了一个在GPU上处理数据的应用程序.代码工作得很好,但我有一个问题,即输入文件(〜3GB,文本)的阅读部分是我的应用程序的瓶颈. (从HDD读取的速度很快,但逐行处理慢).

我使用getline()读取一行,并将第1行复制到一个向量,line2到一个向量,并跳过第3行和第4行.对于11条mio行的其余部分,我们依次类推.

我尝试了几种方法来尽可能地获取文件：

发现最快的方法是使用boost :: iostreams :: stream

其他的是：

>将文件读为gzip,以尽量减少IO,但是比直接慢
阅读它.
>通过读取将文件复制到ram(filepointer,chararray,length)
并用循环进行处理以区分行(也比boost更慢)

任何建议如何使其运行更快？

void readfastq(char *filename,int SRlength,uint32_t blocksize){
    _filelength = 0; //total datasets (each 4 lines)
    _SRlength = SRlength; //length of the 2. line
    _blocksize = blocksize;

    boost::iostreams::stream<boost::iostreams::file_source>ins(filename);
    in = ins;

    readNextBlock();
}


void readNextBlock() {
    timeval start,end;
    gettimeofday(&start,0);

    string name;
    string seqtemp;
    string garbage;
    string phredtemp;

    _seqs.empty();
    _phred.empty();
    _names.empty();
    _filelength = 0;

            //read only a part of the file i.e the first 4mio lines
    while (std::getline(in,name) && _filelength<_blocksize) {
        std::getline(in,seqtemp);
        std::getline(in,garbage);
        std::getline(in,phredtemp);

        if (seqtemp.size() != _SRlength) {
            if (seqtemp.size() != 0)
                printf("Error on read in fastq: size is invalid\n");
        } else {
            _names.push_back(name);

            for (int k = 0; k < _SRlength; k++) {

                //handle special letters
                                    if(seqtemp[k]== 'A') ...
                                    else{
                _seqs.push_back(5);
                                    }

            }
            _filelength++;
        }
    }

编辑：

源文件可在https://docs.google.com/open?id=0B5bvyb427McSMjM2YWQwM2YtZGU2Mi00OGVmLThkODAtYzJhODIzYjNhYTY2下载

由于某些指针问题,我更改了readfastq函数来读取文件.所以如果你调用readfastq,blocksize(行)必须大于要读取的行数.

解：

我找到了一个解决方案,它可以从60秒到16秒的时间读取文件.我删除了内在循环,该循环可以处理特殊字符,并在GPU中执行此操作.这降低了读入时间,只有最小的GPU运行时间增加.

感谢您的建议

void readfastq(char *filename,int SRlength) {
    _filelength = 0;
    _SRlength = SRlength;

    size_t bytes_read,bytes_expected;

    FILE *fp;
    fp = fopen(filename,"r");

    fseek(fp,0L,SEEK_END); //go to the end of file
    bytes_expected = ftell(fp); //get filesize
    fseek(fp,SEEK_SET); //go to the begining of the file

    fclose(fp);

    if ((_seqarray = (char *) malloc(bytes_expected/2)) == NULL) //allocate space for file
        err(EX_OSERR,"data malloc");


    string name;
    string seqtemp;
    string garbage;
    string phredtemp;

    boost::iostreams::stream<boost::iostreams::file_source>file(filename);


    while (std::getline(file,name)) {
        std::getline(file,seqtemp);
        std::getline(file,garbage);
        std::getline(file,phredtemp);

        if (seqtemp.size() != SRlength) {
            if (seqtemp.size() != 0)
                printf("Error on read in fastq: size is invalid\n");
        } else {
            _names.push_back(name);

            strncpy( &(_seqarray[SRlength*_filelength]),seqtemp.c_str(),seqtemp.length()); //do not handle special letters here,do on GPU

            _filelength++;
        }
    }
}

class MemoryMappedFileParser
{
    const char* m_sz;
    size_t m_Len;

public:

    struct String {
        const char* m_sz;
        size_t m_Len;
    };

    bool getline(String& out)
    {
        out.m_sz = m_sz;

        const char* sz = (char*) memchr(m_sz,'\n',m_Len);
        if (sz)
        {
            size_t len = sz - m_sz;

            m_sz = sz + 1;
            m_Len -= (len + 1);

            out.m_Len = len;

            // for Windows-format text files remove the '\r' as well
            if (len && '\r' == out.m_sz[len-1])
                out.m_Len--;
        } else
        {
            out.m_Len = m_Len;

            if (!m_Len)
                return false;

            m_Len = 0;
        }

        return true;
    }

};