OpenHarmony相机用户态驱动框架
作者:润和软件 郭新星
相机作为智能手机上少有的成长空间不错的,能够做出差异化的功能,每年都能成为各大Android手机厂商争相宣传的亮点。众所周知Android采用Linux 作为其内核,而Linux采用的开源协议具有传染性[1],导致Android HAL[2]成为了手机厂商们竞争的重要战场。随着OpenHarmony 3.1[3]的发布,相机模块也逐渐完善起来,目前提供了基础预览和拍照的能力。OpenHarmony中,相机用户态驱动框架承担了和Android Camera HAL一样的角色,这部分位于OpenHarmony的HDF[4]中,对上实现相机HDI[5]接口,对下实现相机Pipeline模型,管理相机各个硬件设备。
相机用户态驱动框架(下图的CameraHost 部分)总体可以分为三层,HDI实现层,实现相机标准南向接口;框架层,对接HDI实现层的控制、流的转发,实现数据通路的搭建、管理相机各个硬件设备等功能;适配层,屏蔽底层芯片和OS差异,支持多平台适配。
模块介绍
HDI Implementation:对上实现HDI接口,向下调用框架层的接口,完成HDI接口任务的转发。
Buffer Manager : 屏蔽不同内存管理的差异,为子系统提供统一的操作接口,同时提供buffer轮转的功能。
Pipeline Core :解析HCS配置完成pipeline的搭建,调度pipeline中的各个node完成流的处理
Device Manager:通过调用底层硬件适配层接口,实现查询控制底层设备、枚举监听底层设备的功能
Platform Adaption :屏蔽硬件差异,为Device Manager提供统一的操作底层硬件的能力
•目录结构
Shelldrivers/peripheral/camera|-- README_zh.md|-- bundle.json|-- figures| `-- logic-view-of-modules-related-to-this-repository_zh.png|-- hal| |-- BUILD.gn| |-- adapter| |-- buffer_manager| |-- camera.gni| |-- device_manager| |-- hdi_impl| |-- include| |-- init| |-- pipeline_core| |-- test| `-- utils|-- hal_c| |-- BUILD.gn| |-- camera.gni| |-- hdi_cif| `-- include`-- interfaces |-- hdi_ipc |-- hdi_passthrough `-- include
•HDI Implementation中的预览流程
接下来我们通过已经发布的OpenHarmony 3.1开源代码,来看看预览是怎么完成的吧
drivers/peripheral/camera/hal/test/v4l2/src /preview_test.cpp存放了针对v4l2的预览测试代码,入口如下
C++TEST_F(UtestPreviewTest, camera_preview_0001){ std::cout << "==========[test log] Preview stream, expected success." <AchieveStreamOperator(); // 获取stream operator // start stream display_->intents = {Camera::PREVIEW}; // 预览流 display_->StartStream(display_->intents); // 起流 // Get preview display_->StartCapture(display_->streamId_preview, display_->captureId_preview, false, true); // release stream display_->captureIds = {display_->captureId_preview}; display_->streamIds = {display_->streamId_preview}; display_->StopStream(display_->captureIds, display_->streamIds);}
先获取stream operator实例
C++void testdisplay::achievestreamoperator(){ // create and get streamoperator information std::shared_ptr streamoperatorcallback = std::make_shared(); rc = cameradevice->getstreamoperator(streamoperatorcallback, streamoperator);// ........}
通过前文的streamOperator创建流
C++void TestDisplay::StartStream(std::vector intents){ // .............................. for (auto& intent : intents) { if (intent == 0) { std::shared_ptr producer = IBufferProducer::CreateBufferQueue(); producer->SetQueueSize(8); // 创建buffer的生产端,并和相应的流进行绑定 auto callback = [this](std::shared_ptr Prebuffer) { BufferCallback(Prebuffer, preview_mode); return; }; producer->SetCallback(callback); streamInfo->streamId_ = streamId_preview; streamInfo->width_ = 640; // 640:picture width streamInfo->height_ = 480; // 480:picture height streamInfo->format_ = CAMERA_FORMAT_YUYV_422_PKG; streamInfo->datasapce_ = 8; // 8:picture datasapce streamInfo->intent_ = intent; streamInfo->tunneledMode_ = 5; // 5:tunnel mode streamInfo->bufferQueue_ = producer; streamInfos.push_back(streamInfo); } else if (intent == 1) { // ....................... } rc = streamOperator->CreateStreams(streamInfos); // 创建流 // ................................ rc = streamOperator->CommitStreams(Camera::NORMAL, ability); // 提交流 // .................................}
下面我们正式进入到hal的源代码中看看是怎么创建流的吧
C++CamRetCode StreamOperator::CreateStreams(const std::vector<std::shared_ptr>& streamInfos){ // ..... for (auto it : streamInfos) {//.... std::shared_ptr stream = StreamFactory::Instance().CreateShared( IStream::g_availableStreamType[it->intent_], it->streamId_, it->intent_, pipelineCore_, messenger_); // 创建流实例// ... StreamConfiguration scg; scg.id = it->streamId_; scg.type = it->intent_; scg.width = it->width_; scg.height = it->height_; PixelFormat pf = static_cast(it->format_); scg.format = BufferAdapter::PixelFormatToCameraFormat(pf); scg.dataspace = it->datasapce_; scg.tunnelMode = it->tunneledMode_; scg.minFrameDuration = it->minFrameDuration_; scg.encodeType = it->encodeType_; RetCode rc = stream->ConfigStream(scg); // 依据上文的流信息配置流// ... if (it->bufferQueue_ != nullptr) { // 绑定前文的生产端 auto tunnel = std::make_shared(); CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel, INSUFFICIENT_RESOURCES); RetCode rc = tunnel->AttachBufferQueue(it->bufferQueue_); CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, INVALID_ARGUMENT); if (stream->AttachStreamTunnel(tunnel) != RC_OK) { CAMERA_LOGE("attach buffer queue to stream [id = %{public}d] failed", it->streamId_); return INVALID_ARGUMENT; } } { std::lock_guard l(streamLock_); streamMap_[stream->GetStreamId()] = stream; // 保存流实例 }// ...}
从上面可以看出,消费端传递到了hal,那必然是由hal从bufferproducer获取buffer,并触发预览的启动流程。那看看AttachStreamTunnel 的实现吧
C++RetCode StreamBase::AttachStreamTunnel(std::shared_ptr& tunnel){ if (state_ == STREAM_STATE_BUSY || state_ == STREAM_STATE_OFFLINE) { return RC_ERROR; } tunnel_ = tunnel; // 绑定生产端 CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR); tunnel_->SetBufferCount(GetBufferCount()); // 配置轮转的buffer个数 TunnelConfig config = {(uint32_t)streamConfig_.width, (uint32_t)streamConfig_.height, (uint32_t)streamConfig_.format, streamConfig_.usage}; tunnel_->Config(config); streamConfig_.tunnelMode = true; return RC_OK;}
CreateStream之后便是CommitStream,这里的CommitStream 做了些什么事情呢,我们接着往下看
C++CamRetCode StreamOperator::CommitStreams(OperationMode mode, const std::shared_ptr& modeSetting){// ...... std::vector configs = {}; { std::lock_guard l(streamLock_); for (auto it : streamMap_) { // 获取流的配置,前文CreateStrea时保存的流 configs.emplace_back(it.second->GetStreamAttribute()); } }// 检查流是否被支持 DynamicStreamSwitchMode method = streamPipeline_->CheckStreamsSupported(mode, modeSetting, configs); if (method == DYNAMIC_STREAM_SWITCH_NOT_SUPPORT) { return INVALID_ARGUMENT; } if (method == DYNAMIC_STREAM_SWITCH_NEED_INNER_RESTART) { std::lock_guard l(streamLock_); for (auto it : streamMap_) { it.second->StopStream();// 如果流被支持,但需要内部重启,这里先停流 } } { std::lock_guard l(streamLock_); for (auto it : streamMap_) { if (it.second->CommitStream() != RC_OK) { // 真正的 CommitStream,下面再细说 CAMERA_LOGE("commit stream [id = %{public}d] failed.", it.first); return DEVICE_ERROR; } } } RetCode rc = streamPipeline_->PreConfig(modeSetting); // 把模式传入进行预配置 if (rc != RC_OK) { CAMERA_LOGE("prepare mode settings failed"); return DEVICE_ERROR; } rc = streamPipeline_->CreatePipeline(mode);// 创建pipeline if (rc != RC_OK) { CAMERA_LOGE("create pipeline failed."); return INVALID_ARGUMENT; } DFX_LOCAL_HITRACE_END; return NO_ERROR;}
C++RetCode StreamBase::CommitStream(){// ... hostStreamMgr_ = pipelineCore_->GetHostStreamMgr(); //从pipelinecore获取hoststreamanager CHECK_IF_PTR_NULL_RETURN_VALUE(hostStreamMgr_, RC_ERROR);// ... info.bufferPoolId_ = poolId_; info.bufferCount_ = GetBufferCount(); // 初始化 bufferpool RetCode rc = bufferPool_->Init(streamConfig_.width, streamConfig_.height, streamConfig_.usage, streamConfig_.format, GetBufferCount(), CAMERA_BUFFER_SOURCE_TYPE_EXTERNAL); if (rc != RC_OK) { CAMERA_LOGE("stream [id:%{public}d] initialize buffer pool failed.", streamId_); return RC_ERROR; } }// stream传递到pipelinecore 并进行绑定 RetCode rc = hostStreamMgr_->CreateHostStream(info, [this](std::shared_ptr buffer) { HandleResult(buffer); return; });// .... return RC_OK;}
CreateStream 和CommitStream结束之后便是Capture,这里包含了起流的动作,关键实现如下
C++CamRetCode StreamOperator::Capture(int captureId, const std::shared_ptr& captureInfo, bool isStreaming){// ...// captureId 捕获请求的id; captureInfo 预览/拍照/录像的参数;isStreaming 连续捕获还是单次捕获(拍照) CaptureSetting setting = captureInfo->captureSetting_; auto request = std::make_shared(captureId, captureInfo->streamIds_.size(), setting, captureInfo->enableShutterCallback_, isStreaming); for (auto id : captureInfo->streamIds_) { // 创建捕获请求,并传递给前文创建的流 RetCode rc = streamMap_[id]->AddRequest(request); if (rc != RC_OK) { return DEVICE_ERROR; } }// ...}
从上面的代码可知预览、拍照、录像都是通过捕获请求触发,单次拍照则为单次捕获请求,预览和录像则是连续捕获请求。
C++RetCode StreamBase::AddRequest(std::shared_ptr& request){ CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR); request->AddOwner(shared_from_this()); request->SetFirstRequest(false); if (isFirstRequest) { RetCode rc = StartStream(); // 起流 if (rc != RC_OK) { CAMERA_LOGE("start stream [id:%{public}d] failed", streamId_); return RC_ERROR; } request->SetFirstRequest(true); isFirstRequest = false; } { std::unique_lock l(wtLock_); waitingList_.emplace_back(request); // 捕获请求添加到waitingList cv_.notify_one(); } return RC_OK;}
看看StreamStream是怎么实现的吧
C++RetCode StreamBase::StartStream(){// ... RetCode rc = pipeline_->Prepare({streamId_}); // pipeline先完成一些准备工作// ... state_ = STREAM_STATE_BUSY; std::string threadName = g_availableStreamType[static_cast(streamType_)] + "#" + std::to_string(streamId_); handler_ = std::make_unique([this, &threadName] {// 创建轮转线程 prctl(PR_SET_NAME, threadName.c_str()); while (state_ == STREAM_STATE_BUSY) { HandleRequest(); // 处理捕获请求 } });// ... rc = pipeline_->Start({streamId_}); // 通知pipeline和底层硬件可以开始出帧了// ... return RC_OK;}
C++void StreamBase::HandleRequest(){ // 如果有 捕获请求下发,则退出等待状态 if (waitingList_.empty()) { std::unique_lock l(wtLock_); if (waitingList_.empty()) { cv_.wait(l, [this] { return !(state_ == STREAM_STATE_BUSY && waitingList_.empty()); }); } }// ... request = waitingList_.front(); CHECK_IF_PTR_NULL_RETURN_VOID(request); if (!request->IsContinous()) { // 如果是连续捕获,则保留一份拷贝在waitinglist waitingList_.pop_front(); } }// 处理捕获请求 request->Process(streamId_);// 最终调用下面的Capture接口 return;}
C++RetCode StreamBase::Capture(const std::shared_ptr& request){ CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR); CHECK_IF_PTR_NULL_RETURN_VALUE(pipeline_, RC_ERROR); RetCode rc = RC_ERROR; if (request->IsFirstOne() && !request->IsContinous()) { uint32_t n = GetBufferCount(); for (uint32_t i = 0; i NeedCancel()) {// 被取消的捕获则退出 CAMERA_LOGE("StreamBase::Capture stream [id:%{public}d] request->NeedCancel", streamId_); return RC_OK; } rc = pipeline_->Config({streamId_}, request->GetCaptureSetting());// 通知pipeline配置 if (rc != RC_OK) { CAMERA_LOGE("stream [id:%{public}d] config pipeline failed.", streamId_); return RC_ERROR; } rc = pipeline_->Capture({streamId_}, request->GetCaptureId());// 这里的capture指的是pipeline中的source node开始回buffer { std::unique_lock l(tsLock_); inTransitList_.emplace_back(request);// 处理过的捕获请求存放在inTransitList } return RC_OK;}
到这起流的流程就结束了,pipeline回上来的帧通过OnFrame接口处理
C++RetCode StreamBase::OnFrame(const std::shared_ptr& request){// ... bool isEnded = false; if (!request->IsContinous()) { isEnded = true; } else if (request->NeedCancel()) { isEnded = true; } { // inTransitList_ may has multiple copies of continious-capture request, we just need erase one of them. std::unique_lock l(tsLock_); for (auto it = inTransitList_.begin(); it != inTransitList_.end(); it++) { if ((*it) == request) { inTransitList_.erase(it);// 已经回帧的请求,从inTransitList删除 break; } } if (isEnded) { // if this is the last request of capture, send CaptureEndedMessage. auto it = std::find(inTransitList_.begin(), inTransitList_.end(), request); if (it == inTransitList_.end()) { std::shared_ptr endMessage = std::make_shared(streamId_, request->GetCaptureId(), request->GetEndTime(), request->GetOwnerCount(), tunnel_->GetFrameCount()); CAMERA_LOGV("end of stream [%d], ready to send end message, capture id = %d", streamId_, request->GetCaptureId()); messenger_->SendMessage(endMessage); pipeline_->CancelCapture({streamId_});// 如果此次捕获结束,则取消捕获 } } } ReceiveBuffer(buffer);// 底层返回的buffer送还到生产端,最终帧数据送到消费端 return RC_OK;}
附录
-
linux和Android的关系 - 知乎 (zhihu.com) ↩︎
-
HAL Subsystem | Android Open Source Project (google.cn) ↩︎
-
zh-cn/release-notes/OpenHarmony-v3.1-release.md · OpenHarmony/docs - Gitee.com ↩︎
-
OpenHarmony HDF 驱动框架介绍和驱动加载过程分析-OpenHarmony技术社区-51CTO.COM ↩︎
-
OpenHarmony HDF HDI基础能力分析与使用-51CTO.COM ↩︎