In Peter's House, "Assalam Alaykum"
The three-day forum opens "a new chapter in the long history" of dialogue between the two faiths, the head of the Catholic delegation, Cardinal Jean-Louis Tauran, told the French Catholic daily La Croix.
Benedict will meet with the delegations on Thursday.
The Muslim side is led by the mufti of Bosnia, Mustafa Ceric, whose spokesman Yahya Pallavicini told AFP the delegates "represent no state and no ideological tendency."
The delegation includes Swiss intellectual Tariq Ramadan, an outspoken and controversial Muslim figure in Europe, along with Aref Ali Nayed of the Islamic Centre of Strategic Studies in Amman, Jordan, and Iranian ayatollah Seyyed Mustafa Manegheg Damad.
Several women in the delegation include Ingrid Mary Mattson, a professor of Islamic studies at the Hartford (Connecticut) Seminary in the United States.
The Vatican seminar was organised in response to a Muslim call for dialogue issued in October 2006, a month after Benedict's speech in Regensburg, Germany.PHOTOS: AP/Hussein Malla(1); Salih Zeki Fazlioglu/Anatolia(2)
The call titled "A Common World" was signed by 138 Muslim religious figures and scholars.
"Now that the shock waves touched off by Pope Benedict XVI's remarks ... have subsided, the overall consequences have proven more positive than negative," Ramadan commented in the British daily The Guardian's online edition.
"Above and beyond polemics, the pope's lecture has heightened general awareness of their respective responsibilities among Christians and Muslims in the West," he added.
The Regensburg lecture sparked days of sometimes violent protests in Muslim countries, prompting the pontiff to say that he was "deeply sorry" for any offence and to attribute Muslim anger to an "unfortunate misunderstanding."
The closed-door discussions at the Vatican will focus Tuesday on "God's love" and Wednesday on "loving your neighbour," a theme that touches on two Vatican priorities, human rights and religious freedom.
The Vatican is however cautious over opening a purely theological dialogue, with Tauran telling La Croix: "We'll see ... how far we can go together."